in this case you wont be able to use the calcualtion. You will need to use a Discount rule for each quantity break.
In the discount rule, use the trigger "Number of Items With a specific value", you will need to know the ID of the registration Item.
for example, the discount for $5 discount off of each registration when you register 3-5 people at the same time would be set up as:
Triggers:
Number of Items With a specific value:
If the total number of items where "ID" has a value of "22" is ">=" a value of "3"
AND
Number of Items With a specific value:
If the total number of items where "ID" has a value of "22" is "<=" a value of "5"
Calculations:
Based on items With a specific value:
Total of "Quantity" where "ID" has a value of "22" "Times" "5"
this example assumes the ID of the item is 22. The other discount rules will be similar, just changing the quantity breaks and the multiplier.