Sorry, its friday night here in the uk, so i only had a chance for a quick look.
One thing strikes me, are the groups of numbers you sell always from the beginning of a particular block of a hundred ?
eg 101 or 201 or 301 ? or could they be random, 237 to 336 ?
if the former, you could possibly have a form like this...
001 - 100  sets
101 - 200  sets
201 - 300  sets
where the customer enters the quantity for each number set
you would have to work out how to add multiple rows to your cart at once, but at least you would have a defined quantity of sets for your pricing and shipping calculations.
I would imagine the above form would be available for the more 'custom' orders, a simpler group of options could be displayed by default perhaps?
if i am wide of the mark here, please let me know and i will take a proper look.