The only was to get it to scroll up or down is by using the scroll button while holding the left mouse button down.
That is how it works. It doesn't auto scroll the page when you drag it to the top or bottom. A workaround is to use the scroll button like you said, or drag and release the image as far down as you can go, scroll the page, then continue dragging it again. Also, there is the image size toggle button which will show more images in the grid. That might help you as well.