You should be able to use a subdomain when activating Cafe Commerce, make sure you have the sub=domain registered in your Dreamhost control panel before trying to register it with cafe commerce.
The error that the domain is already taken happens if the domain already has a DNS to another IP address, also in the Dreamhost control panel, you should be able to release the domain to allow cafe commerce to use it. If you need help with the Dreamhost control panel, contact Dreamhost support, they will be able to help with the control panel settings.