It would take custom coding to pull this off. It isn't going to be covered by a rule.
You'd have to add php code to loop through the cart, find the most expensive item less than the promo item and then create a session variable to store the amount.
The rule would just discount the amount stored in the session variable... but you'd need custom code to determine how much. It is only a handful of lines of php code, but you'd need to be able to write php to loop through an array and set the session variable by hand.