you could use eCart and Data Bridge for creating this type of system.
I think you would want to use security assist to create a login system. once the user is logged in, you can lookup their account to determine their outstanding balance.
in the eCart add to cart button, you could set the Price to be updatable from a text box, this would allow them to enter the amount they wanted to pay.
after the order has been processed, you would need to update their balance using the update record behavior to subtract the amount that was paid.