This is a generic error message that your webserver is returning. To determine what the problem is a more meaningful error message will be needed. To enable the more advanced error messaging in IIS 7 for Vista you will need to do the following:
1) Go to Control Panel > Administrative Tools
2) Open the IIS control panel and open the ASP option
3) At the top of the ASP control area select Display: Friendly Names
4) Now go down to the Compilation area and expand Debugging Properties
5) In debugging properties make sure to set Enable Client -side Debugging, Enable Server -side debugging, and Send Errors to Browser to TRUE. Make sure to click the Apply on the right side of the page.
6) After setting the debugging you must go back to the IIS control panel and using the controls on the right side restart the server.
This will enable debugging info to display when an error occurs. This should be turned on while testing and should not be left enabled for a live server.
One thing I can see that may be causing the issue is that the checkout form does not include the shipping fields. Protx requires shipping information to be passed, You should delete you checkout pages and re run the checkout wizard. Leave the will you offer shipping services option checked. On the next page, you can leave the Shipping provider set to none and the Base Rate and Incremental Rate set to 0.
If you still have problems, we will need to know the actual error that is occurring to be able to offer assistance. You may need to contact the server to have them turn on error reporting, or look into the error logs to see what is going wrong.