It's said, in annoyance, of many a pest (4)
I believe the answer is:
drat
'it's said in annoyance' is the definition.
I can't tell whether this defines the answer.
'many a pest' is the wordplay.
'many' becomes 'd'.
'a pest' becomes 'rat' (rat can mean a nasty person).
'd'+'rat'='DRAT'
'of' acts as a link.
Can you help me to learn more?
(Other definitions for drat that I've seen before include "Expression of annoyance on DART" , "Minced oath" , "Bother" , "Expression of mild annoyance" , "Curse it!" .)