Fancy being with that man (4)
I believe the answer is:
whim
'fancy' is the definition.
(I've seen this before)
'with that man' is the wordplay.
'with' becomes 'w' (abbreviation).
'that man' becomes 'him' (I've seen this before**).
'w'+'him'='WHIM'
'being' is the link.
(Other definitions for whim that I've seen before include "Sudden unusual change of mind" , "Fanciful impulse" , "Capricious desire" , "Urge" , "Fad" .)