now since I set the session variables on the checkout page (since this is where the customer types in their address details) 
normally when using express checkout, it would be done in conjunction with Direct Payments.
on the checkout page, there is normally a checkout with paypal express button, then a separate form where the user can enter their information to checkout with Direct Payment, this second form passes to the confirm page to set the session variables.
The checkout with express checkout form would only contain the express checkout button and does not send any of the information that would be entered, it not supposed to.
The user would go to paypal, enter the billing information, then this information is sent to the pp_confrm page, only on the pp confirm page are the sessions set.
you are correct that there are 3 ways for the failure to happen:
1) on sending the user to the paypal side - this is why we need the PayPal Authorize Express Checkout bindings
2) directing from paypal to your site - this is why we need the Get Payer Profile from PayPal Express Checkout
3) On processing the order - this is why we need the PayPal Process Express Checkout bindings
on the failures that are happening where there are no session set, that would be happening on either passing the user to the paypal site, or on getting the payer profile.
the failures that happen where the sessions are set will happen on processing the transaction.
