Sounds like you are trying to use a MySQLi connection with the standard DW Recordset dialog box or vice versa.
If the Connection is a Dynamic MySQLi connection, use the MySQLi recordset dialog that is under webassist -> MySQLi -> MySQLi recordset.
Your friends over here at WebAssist! These Dreamweaver extensions will assist you in building unlimited, custom websites.
These out-of-the-box solutions provide you proven, tested applications that can be up and running now. Build a store, a gallery, or a web-based email solution.