Reminding you of your transgressions is irritating (4,5)
I believe the answer is:
hair shirt
If this is the answer I cannot explain how the clue works.
Can you help me to learn more?
(Other definitions for hair shirt that I've seen before include "Penitential garment of hirsute kind" , "uncomfortable clothing" , "Traditional wear of ascetic" , "Austere garment" , "Something worn by penitents" .)