When you are getting the invalid file type message please describe exactly what you are doing and where you are at when this occurs.
You should be able to use the link button to insert a link to a page. If the page already exists on the server you should be able to select 'Browse Server' to select the file. What happens for you when you attempt to browse the server?
You can also just add in a link yourself to the page by changing the protocol to <other> and just specifying the relative path to the page from where this page will be served from on the site.