Not initially a polite address to woman or man (4)
I believe the answer is:
adam
'man' is the definition.
(man's name)
'not initially a polite address to woman' is the wordplay.
'not initially' means to remove the first letter.
'a polite address to woman' becomes 'madam' (I've seen this before).
'madam' with its initial letter removed is 'ADAM'.
'or' acts as a link.
(Other definitions for adam that I've seen before include "the first orphan?" , "First man of Genesis" , "First fruit-eater" , "Early man" , "One of our common ancestors?" .)