What is the custom site URL? you can post that in a private message if you like.
It sounds like the custom site may not have an SSL certificate, if there is no SSL certificate for the custom domain, it will redirect to the secure cafe commerce domain for pages that require ssl like registration and checkout.
If we know your sites UR we can double check this to make sure that is the problem and not something else.