An irritating reminder of past misdemeanours (4,5)
I believe the answer is:
hair shirt
But I can't explain it at all! Could you help me understand?
(Other definitions for hair shirt that I've seen before include "Hirsute penitential garment for men" , "one's ashamed to wear it" , "article for an ascetic" , "Penitential hirsute garment for upper part of man's body" , "A penitential garment of hirsute beginnings" .)