Art set in amalgam, a rectangular arrangement (6)
I believe the answer is:
matrix
'a rectangular arrangement' is the definition.
(mathematical term)
'art set in amalgam' is the wordplay.
'set' indicates an anagram (instruction to set the letters).
'in' indicates putting letters inside.
'amalgam' becomes 'mix' (an amalgam is a mixture).
'art' anagrammed gives 'atr'.
'atr' going inside 'mix' is 'MATRIX'.
(Other definitions for matrix that I've seen before include "Mathematical array; The --, sci-fi film" , "A maths array in rows and columns" , "network of circuit elements" , "Mathematical arrangement of rows and columns" , "Mould; array (maths)" .)