Fancy wife being associated with that man (4)
I believe the answer is:
whim
'fancy' is the definition.
(I've seen this before)
'wife being associated with that man' is the wordplay.
'wife' becomes 'w'.
'being associated with' means one lot of letters go next to another.
'that man' becomes 'him' (I've seen this before**).
'w'+'him'='WHIM'
(Other definitions for whim that I've seen before include "Impulse or caprice" , "Capricious desire" , "'Caprice, notion (4)'" , "Sudden impulse" , "Caprice, sudden fancy" .)