Maybe borderline books with note written in (5)
I believe the answer is:
limit
'maybe borderline' is the definition.
'books with note written in' is the wordplay.
'books with' becomes 'lit' (abbreviation for literature. I am not sure about the 'with' bit.**).
'note' becomes 'mi' (as in 'do re mi').
'written in' means one lot of letters goes inside another.
'lit' placed around 'mi' is 'LIMIT'.
(Other definitions for limit that I've seen before include "Boundary or restriction" , "Extreme point" , "Constraint" , "Place bound on" , "Restrict - boundary" .)