I'm not familiar with the modal box you are referring to and will not be able to offer specific support on that part of it. Do you have the user login pages and protected pages created already?
If so it sounds like you could just use the modal control to load your login page instead of going to the login page directly. To get the user to go to the fail page you can adjust the authenticate user server behavior to set the failed redirect. On the login page you can see this as:
"failRedirect" => "<your failed page>.php",