There are varying levels of security you can have your account configured to. You may or may not be required to have a matching street address in order to accept a card payment.
Removing the validation would require:
1) removing the word "required" from the form input in the HTML itself
2) search on the cart_confirm.php page and find references to that field in the server validation and delete the lines where it is referred to.