You could probably use javascript in the show if area to re-open the modal if it fails validation.
Keeping the modal open is more difficult. It would mean doing validation through AJAX, and would require more hand coding. I could help with that in a premier support ticket.
The success message can be wrapped in a server validation show if region to only show if all validations pass. That is the easier solution.