It’s silly to exclude me from the game again! (4)
I believe the answer is:
Here is my best explanation:
'it's silly' is the definition.
'me from the game again' is the wordplay.
'from' indicates named letters should be taken away.
'again' becomes 'ga' (I can't explain this - if you can you should believe this answer much more).
'game' with 'me' taken away is 'ga'.
'to exclude' acts as a link.
I am not very happy about this link. Some or all of it may be part of another bit of the clue.
(Other definitions for gaga that I've seen before include "person who is senile?", "Repetitive slang for senile or crazy", "Slangily crazy, off one's head", "US singer Lady ___", "Lady who sings", "bananas", "slightly mad", "In repetitive slang, senile or crazy", "Barmy joke, third in cracker" and "fatuous?".)