I had this problem earlier, and it seemed upgrading to the newest ckeditor fixed it, however it did not.
I can sometimes get it to open the filemanager in a new window, by changing some settings (ie compatability mode, popup blocker etc), and I can get it to work for a while, then If I clear my cache, and then open ie again, the problem returns.
Sometimes then I can change settings again (turn popup blocker on, when I had turned it off before, toggle compatability mode etc), and sometimes it will work again for a while.
I have been unable to get it to consistently open the filemanager in a new window though, in IE9 (all other browsers work fine).
It is very odd, because it seems intermittent.
Tom
