I think you are going to have to use a combination of calculations and rules to pull this off.
I think you could add calculations:
qty_discount = (([Quantity]>=5)?([Price]*0.3):0) + (([Quantity]>=10)?([Price]*0.2):0)
Then your calculation for the discount rule would just be the subtotal of the column "qty_discount"
This would give 30% off for 5-9 and 50% off 10 or more.