Very brief affair involving Tory's ending... (6)
I believe the answer is:
flying
'very brief' is the definition.
(as in a flying visit)
'affair involving tory's ending' is the wordplay.
'affair' becomes 'fling' (a fling is a romantic affair).
'involving' means one lot of letters goes inside another.
'ending' suggests the final letters (the ending of the word).
The last letter of 'tory' is 'y'.
'fling' placed around 'y' is 'FLYING'.
(Other definitions for flying that I've seen before include "Taking to the air" , "Journeying through the air, maybe with Ryanair" , "(Of flag) displayed on a flagpole" , "running fast" , "Getting high" .)