There was a discussion on Cloudy Nights last February about optimal settings for mode 1 (high gain mode). The consensus was that Gain 56 was a good choice. Qiu Hongyun from QHY stopped by and recommended it for short exposures. I have been using Gain 56 with offset of 10 since I got the camera, and the results are excellent.
https://www.cloudynights.com/topic/754530-qhy268m-read-out-modes-questions/
I also have Baader 36 mm filters, so your setup has some similarities to mine. The camera and filters are being used with a Sky-Watcher Esprit 100ED. One issue I have encountered is halos around brighter stars in RGB images, especially the blue stars. I need to remove halos with software every time I process an image. I cannot say for certain that the filters are responsible, but I think it is likely. The Baader H-alpha filter performs well, with minimal halos, and the red filter is also reasonably good. It's the green and blue filters that have halo problems with the optics I am using.
Thanks for your reply and for the link to CN.
Yes, according to the diagrams on the QHY-website, gain 56 for mode 1 and gain 26 for mode 0 are the optimum starting points. And gain 60, mode 1 is the best point regarding the dynamic range, as Stjepan uses.
Well, as usual, the only imaging, I can do with a new camera, is doing darks, biases and flats...
Do you use the new generation Baader "CMOS-optimised" filters? They should not have that strong halos at all!