Well, that's a tricky one.
you cant really do a simplistic discount calculation of
[subtotal] - [all discounts] * [amount]
because as soon as that discount is applied the value of [all discounts] will be changed meaning the discount will be recalculated in an endless loop.
instead, you will need set the caculation for the discount to subtract each other discount from the subtotal
For Example, if I have 2 discounts:
Discount 1
Discount 2
i can set Discount 2 to subtract Discount 1 from the subtotal using a flat rate discount and the following code:
($this->TotalColumn("TotalPrice") - $this->GetRuleValueByName('Discounts','Discount 1')) * 0.1
this will give a 10% discount of the Sum of (Subtotal - the value of Discount 1)