According to the Authorize.net documentation, the address info is required under the following circumstances:
Required when using GPN Canada or WorldPay Streamline Processing Platform.
For more information on AVS, see the Merchant Integration Guide at http:// www.authorize.net/support/merchant/
so it looks like it should be OK to remove those fields if you disable the Address Verification Service security feature in your account.