It's the fellow I am scared of, not a young female (6)
I believe the answer is:
heifer
'a young female' is the definition.
(young female cow)
'it's the fellow i am scared of' is the wordplay.
'it's the fellow' becomes 'he' (pronoun for a man).
'am scared of' becomes 'fer' (I am not sure about this - if you are sure you should believe this answer much more).
'he'+'i'+'fer'='HEIFER'
'not' is the link.
This may not be right. It may be part of another bit of the clue.
Can you help me to learn more?
(Other definitions for heifer that I've seen before include "part of stock" , "Female bovine that has not given birth" , "Young cow which has not yet given birth" , "beast" , "a little lower" .)