The way you would do it is you would add a recordset filtered with the form elements you want to be unique. That way the recordset will either be empty (no repeats) or have a result (repeat).
then you can make sure those recordsets appear before the validation toolkit and use server validation to ensure the number of rows in that recordset is equal to 0.
The exact syntax depends on your server language.