you would need to have a Cart column for the category and store the category in that column using the add to cart buttons.
instead of using a calculation for this, you would use a discount rule on the discounts tab.
Set the trigger to:
"Number Of Items with A Specific Column value"
if the number of items where "category" has a value of "category name" is ">=" a value of "3"
and in the calculations section, use:
"Based on items with a specific value"
Total of "Total Price" where "category" has a value of "category name" "times" "0.10"