The extension manager is extremely slow when removing extensions unfortunately. And even more unfortunate is that we can't really do anything to improve it.
The extension manager is relatively quick when installing extensions, so one work around is to rename your user configuration folder, which will effectively uninstall all of your extensions at once immediately, then reinstall any extensions you need.
Reinstalling 20 extensions can be faster than uninstalling or updating 1.