don't do this on the calculation tab, as you have discovered, calculations are applied on a line item basis.
Instead, do this on the shipping tab.
Create a new shipping rule:
based on number of unique items in the cart > 0
Flat Rate calculation of:
(($this->TotalColumn('TotalPrice') * 0.1) > 4)?$this->TotalColumn('TotalPrice') * 0.1:4