Mathematical subgroup (5)
I believe the answer is:
coset
'mathematical subgroup' is the definition.
(I've seen this in another clue)
This is all the clue.
(Other definitions for coset that I've seen before include "subgroup" , "group in maths class" , "mathematical group" , "mathematicians' group" .)