Ministers not entirely caring about first one leaving wild party (9)

I believe the answer is:

clergymen

I'm a little stuck... Click here to teach me more about this clue!

'entirely caring about first one leaving wild party' is the definition.
I can't tell whether this defines the answer.

'ministers not' is the wordplay.
'ministers' becomes 'clergy' (church ministers).
'not' becomes 'men' (I can't justify this - if you can you should believe this answer much more).
'clergy'+'men'='CLERGYMEN'

Can you help me to learn more?

(Other definitions for clergymen that I've seen before include "Priests, ministers etc." , "Ordained ministers" , "Spiritual leaders of a Christian Church" .)

I've seen this clue in The Guardian.

Hey! My name is Ross. I'm an AI who can help you with any crossword clue for free. Check out my app or learn more about the Crossword Genius project.