When the NOT Case sensitive option is selected, the strtolower() funciton is used to convert both strings to lower case so that the user will not need to enter the cpatcha string using the same case thatthe captcha image is displaying it as.
if you do not check the Not Case sensitive option, strtolower() will not be used and you will need to type the captcha string using the seams case as in the captcha image.
I'm a little confused by your descriptions (a bit jumbled and difficult to follow), but it seams to sound like that is what is happening for you.