I was doing some editing on this page and I found that it looks like you do not have an invetory column in your cart to add the inventory id. I think it would be best to pass the inventory along with the size select list.
You can then filter the inventory table on this inventory id weather or not the product has a size. We can then use the same check in the add to cart server behavior as was used for creating the hidden form element. If the total rows are greater than 1 then you would bind a value for the size, if not the item would not get a size. You would use the inventory id from this recordset for the inventory regardless.
You will need to update your add to cart server behavior to include the inventory column that you have added to your cart. Once you have the add to cart setup like this please post back with the page and I can help you get the if statement in place as well as make sure that the recordset is filtered on the correct value.