Instead of removing the one validation from the Server Validations, you should remove the server validations server behavior in this case.
in the server behaviors list, select the server validation server behavior, then click the minus button to remove it.