the gallery uses image resizer to create a new copy of the image at the dimensions specified for the large image in the gallery settings.
in the webassist/caching/CachedResize_Image.php
you can change line 207:
$process = $image_save_func($image_c, $save_path);
$process = $image_save_func($image_c, $save_path, 100);
to force the resized to image to be created at the highest possible clarity for a jpeg