I'm not sure why your email takes so long to process. I sped up the site with some php trickery.
The first thing I tried is:
1) I updated the email code to the latest version (that didn't help)
2) I updated the contact page to submit directly to the thankyou.php page (that didn't help either).
3) I moved the email to a different page and hand wrote php code to use curl() to post to the new email page and set it to not wait for a result.
#3 ended up being the solution that worked. I could have undone #1 and #2 but they didn't make a difference either way so I just left them.