You should be able to just add another condition to your existing shipping charge rule. The condition will be like this:
'Number of items with a specific column value'
If the total number of items where [category] has a value of [your category] is < a value of 3.
When you add this condition to your shipping charge rule it will ensure that shipping is only applied when this condition is not met.