Work out nasty unknown rules of grammar (6)
I believe the answer is:
syntax
'rules of grammar' is the definition.
(I've seen this before)
'work out nasty unknown' is the wordplay.
'work out' is an anagram indicator.
'unknown' becomes 'x' (algebra).
'nasty' with letters rearranged gives 'synta'.
'synta'+'x'='SYNTAX'
(Other definitions for syntax that I've seen before include "Sentence structure" , "Grammatical construction" , "Way in which words are arranged to form sentences" , "The rules governing the arranging of words" , "The rules governing words arrangements" .)