I'm not so sure that the file permissions would have this effect. If there is a script that is attempting to change permissions on a file it should not be able to set these permissions if they are not allowed.
If you are getting the file with 777 then it might be because of the folder that it is uploading to. There is a setting for the default permissions for uploaded files, it can be found at:
HTMLEditor > editor > plugins > kfm > configuration.php
In this file near line 194 is the default file permissions that are set. Can you upload anything other than an image and get this same result? When you say that it hangs does it look like it is waiting for a response, or trying to connect to a page or anything like that?