I'd suggest doing this in a premier support ticket.
You would have to set session variables to keep track of the coupon and the category it applies to and the coupon rate. Then you could probably create a calculation in the cart for DiscountedItems which is a sum of the TotalPrice of any cart rows that are in the coupon category.
Then add a discount rule that calculates the correct discount based on the DiscountedItems calculated column total.
It is a bit complex, that is why I'd suggest a premier ticket if you have it in the budget.