There is a fundamental problem with this scenario, you only validate values for form elements, not weather they are on a page. So to do this server side you would need to check the value of the element, if it is not there this will cause the validation to fail.
One way to get around this would be to apply the server validation and set the validation redirect to go back to this same page. This will work fine for the items with size and color, for the items that do not have a size and color though you would need to add in some hidden form elements named the same and have a value for them. With these items though you want to make sure you do not bind a size and color in the add to cart server behavior otherwise it will use these hidden form elements.