Island recluse is not with it (4)
I believe the answer is:
herm
'island' is the definition.
'recluse is not with it' is the wordplay.
'recluse' becomes 'hermit' ('hermit' can be a synonym of 'recluse').
'is not with' suggests deleting specific letters.
'hermit' with 'it' taken out is 'HERM'.
(Other definitions for herm that I've seen before include "One of the Channel Islands" , "bust" , "Island in English Channel" .)