If you have the recordset already in place on the page then you could use server validations rather than hand coding the header and the custom error message.
You are definitely going about this correctly, I think if you use the server validations for this part you will have better luck. You need to add to the server validations that are already on the page or add a new one.
For the validation choose the same trigger as your insert, then set it up like this:
validation type: number
server variable: $totalRows_<your email checking rs>
add it like this, then configure the validation like so:
format: English (1,000.00)
allow decimals: all
I'm including a screen shot of the number information for you as a reference. Once you have configured the validation you can specify the same page as the failed redirect. The last part of this is to add in a error message to let the user know that they can't use that particular email address. With the message crafted select it and go to Server Behaviors > Validation Toolkit > Validation Show If to make it only display when the validation fails.