you have to have 2 rules set up.
one of the flat rat and another for the shipping rate lookup.
you need to have a column in the cart to mark whether they should be applied to the flat rate rule or the shipping rat lookup rule.
For example, lets call that column:
shipMethod
for flat rat items set that columns value to 1 for rate lookup set that column to 2
in the shipping rules add triggers to look at the value for that column
to get this working for the rate lookup will require some hand coding, you would need to add code to calculate the weight for the items that have shipmethod set to 2. We can help implement that part in a premiere support appointment:
http://www.webassist.com/premier_request.php
to troubleshoot the checkout process, add the following code to the checkout failure page to dump the session:
<pre><?php var_dump($_SESSION); ?></pre>
the session contents will contain the response from the gateway, this will have information on why the transaction is failing.
