Gain settings for Flats vs Lights

The gain needs to match, the temperature does not. The temperature of bias or flat darks used to calibrate the flats needs to match the flats themselves.

In general I take everything the same to be safe.

Insofar gain may influence PRNU then you should keep the same gain as is for lights. IOW, it depends on the sensor. So far I haven't seen a compelling reason to keep them the same on my sensors.

Rick Veregin:
I never got good consistent flats until I started using the same gain as the lights, and using temperature/time match darks and dark flats.

This is good practical feedback to have. Is there any way you can isolate the effect of 'same gain as the lights' vs the other two, perhaps with an example? Temp/time matched darks and dark flats are obvious requirements for good calibration. It is the matched gain that confuses me.

It seems to me that you can easily figure out whether gain is a factor or not by taking two sets of bias frames at different gains, stack them up and then create a frame as the difference between these master bias frames. If the PRNU is the same you will end up with a constant-valued frame (minus residual noise level which can be minimized by taking 50 frames or so for each master bias).

Correction: replace the word bias with the word flat.

Tim Hawkes:
I am in the no camp  (basically because I have never detected a particular problem with flats as I make them now).  It is worth reading Robin Glover on the sharpcap forum on this topic.  His analysis appears entirely logical to me anyhow


My understanding is 

1) For flat frames you only need to measure the relative amount of light falling on each part of the sensor  so no need for flats to match the gain of lights

2) Thus it is important to record flats at a level where the response is linear (so not close to LH axis and neither above say 80% saturation?)

3) For artificial light sources the exposure time also needs to exceed that of any flicker

So generally the specific gain used for flats doesn't matter


But there are also some important camera-specific aspects to this question.  There is - for example - apparently a specific issue for example with 294 sensors and for example in ASI 294 MM and MC cameras wherein at certain gain levels some small number of the pixels fail to saturate whatever the exposure time.   That means for example that it is recommended not to make flats within a certain range of gain settings 120-200  (~ 8.7 dB I think) if you use these cameras.

Tim

I'd disagree with statement in 1) in that you need to measure the relative response to a uniform illumination of the sensor (and this begs the question on how uniform is your flat panel? But I'm digressing here...), which might or might be not dependent on gain . As for the issue with gain issues in a ASI294MC Pro I can only say that it has NEVER been the case and I used it with 120 gain flat out for nearly 3 years now.

Practically, I doubt it makes much of a difference and for the limited tests I've run I can't see any meaningful difference in the output. Theoretically, that is a different question which is best answered empirically, by running a test as suggested in my replay above yours.

Tim Hawkes:
It's a relative response because you subtract the dark and divide by the flat in calibration?

The issue with the 294 does appear to be  real Andrea  - that is hidden somewhere on the ZWO website and relates to the Sony chip  - I have only just become aware of it myself.  It may of course be that the actual effect of the problem on flats is subtle or not easy to notice - I don't know.   

It is very easy to test anyway.  Hook up the camera set it to gain 130 or whatever (into HCG mode) and increase exposure while observing the saturation response.  Apparently with our chip -- I have the same -  even  at exposures well 'above' saturation' a few percent of the pixels will still be showing on the graph at around 70-80%.

It's a relative response because, in principle you don't know the effective QE of each pixel and the relative shading each pixel is subjected to.

Willem Jan Drijfhout:
Thank you for your extensive response, Rick. And by the way you have an impressive portfolio of long exposure images!
Your first argument points in a similar direction as Andrea's, that the PRNU fixed pattern noise changes with gain. PRNU itself is a pixel-level difference in response to gain, which causes the fixed pattern of non-uniformity as a kind of fingerprint of the sensor. The question is then if the pattern/fingerprint is different at different gain levels?

The other point you bring up is about offset (or bias as you call it). So perhaps the idea of keeping gain constant is about keeping offset constant? Offset is meant to avoid any 0-pixels and essentially avoid dark clipping. This is important for bias/dark frames and should therefore be the same in any frame you calibrate with them. But once properly calibrated it is hard to see what further effect it can have. Especially on flat frames that typically have 25-50% illumination. On another note, with modern cameras it is not too difficult to keep offset the same within the normal range of gains used. I've tested this for both the ASI533MM and the QHY268c, and both have such clean signal that a fairly low similar offset for both typical gain settings avoids any clipping.

Going the safe route makes a lot of sense, but there will still be that itch of wanting to know why we do it. And for me it has practical consequences. To get to let's say 30% illumination, there are three variables to work with: light intensity, gain and exposure time. So far I used a flat cap that allowed automated fine adjustment of brightness per filter in software. As I'm preparing for a bigger OTA, the options for flat panels become less. And light intensity can only be manually adjusted in software or by adding additional diffusers/pieces paper etc. So that leaves only gain and exposure time to play with. And if gain is not an option, it's only exposure time which will likely lead to some very long exposures. Anyway, rattling here a bit. Nothing that can be overcome, it just got me into this rather fundamental question.

Yes, I am saying the same thing as Andrea. The answer is it might be affected by gain, or better, there is no reason to expect the fixed noise must be identical.  Though possibly it might be good enough for your camera sensor and your images. 

Yes, it is a bias or offset signal added to all pixels, and results in a bias frame when you use a short exposure. Yes, if you set your offset high enough you can run with that for all gains, only downside is you lose a bit of dynamic range potentially, but generally the offset is small enough that it doesn't matter. This is in fact what I do now, I am using a higher offset at higher gain, so if I drop gain I leave the offset where it is. I assume other cameras are similar. This would yes remove my reservation about the different offset, but not the potential effect of pattern noise. Not sure what your question about calibration is. Correct calibration is:

CalibratedLight = (Light - Dark) / (Flat - FlatDark)

The offset/thermal dark noise in either case is not a light signal. Flats are used to map the non-uniform light response, so if you don't subtract the respective darks from the lights and flats you will not get the correct light signal correction. Some people get away with this if there is no vignetting, but I have an SCT which cuts into my APS-C field, if I don't do the darks absolutely right (light and dark match settings, while flat and flat dark match each other) then my field is not properly corrected when I stretch. As I mentioned I find gain matching light and flats works better too. My camera is very linear response over the range I use (SharpCap has a nice routine to semi-automatically work you through testing out your camera--you can determine linearity, read noise, etc), so exposure differences for flats do not affect the linearity compared to lights. As for noise, remember each flat has the same read noise as a light. The good thing as you point out is that the flats are divided (or multiplied) depending how your SW handles them, which means the S/N is what is important. And as you say the signal is strong. However, if you do hundreds of images and only 10 flats, the noise in the flats can contribute. Note as well fixed pattern noise is not averaged in flats, so you do not want the fixed pattern noise to be different in flats and lights, it will show up (dithering your lights helps this, as it moves the target relative to the flats, averaging out the pattern noise--but better not to have it to begin with).

I use an artists A3 flat panel from Amazon with my 9.25" SCT, probably this would be good up to 10 or 11" scope, and was only about $50 if I recall. It also has 3 light settings, so is adjustable. I set it at the lowest value for high gain then adjust exposure. Uniformity from it has never been an issue. My own flats are usually a fraction of a second at high gain. At low gain I turn up the light. Even running one hundred flats is done is less than a minute. The RASC robotic 16" scope uses just sky flats, works well, though each exposure may need to be slightly different unless you can take them all very quickly one after the other.

My advice always is integrate advice, be wary of absolute NO/YES statements with no reasoning, the theory is around if you look for it, and put it all together as best you can, and then test it yourself. Note measuring noise accurately is not that easy. So for example, take a good set of lights so you noise will be low. Then use just a few flats a few darks and change things up. Stretch all images the same exactly, then see what noise and flatness look like. Come to your own conclusions of what affects what for you. Taking extra flats and darks with varying conditions is not a lot of work to understand your system--and there are many cloudy nights to experiment. It is worth it, the unfortunate fact, that is not often talked about, is the longer your total exposure and the better the image data, the better your flats and darks need to be, and the more of them you need if you really want to get all you can out of that data.
Rick

Rick Veregin:

Willem Jan Drijfhout:
Thank you for your extensive response, Rick. And by the way you have an impressive portfolio of long exposure images!
Your first argument points in a similar direction as Andrea's, that the PRNU fixed pattern noise changes with gain. PRNU itself is a pixel-level difference in response to gain, which causes the fixed pattern of non-uniformity as a kind of fingerprint of the sensor. The question is then if the pattern/fingerprint is different at different gain levels?

The other point you bring up is about offset (or bias as you call it). So perhaps the idea of keeping gain constant is about keeping offset constant? Offset is meant to avoid any 0-pixels and essentially avoid dark clipping. This is important for bias/dark frames and should therefore be the same in any frame you calibrate with them. But once properly calibrated it is hard to see what further effect it can have. Especially on flat frames that typically have 25-50% illumination. On another note, with modern cameras it is not too difficult to keep offset the same within the normal range of gains used. I've tested this for both the ASI533MM and the QHY268c, and both have such clean signal that a fairly low similar offset for both typical gain settings avoids any clipping.

Going the safe route makes a lot of sense, but there will still be that itch of wanting to know why we do it. And for me it has practical consequences. To get to let's say 30% illumination, there are three variables to work with: light intensity, gain and exposure time. So far I used a flat cap that allowed automated fine adjustment of brightness per filter in software. As I'm preparing for a bigger OTA, the options for flat panels become less. And light intensity can only be manually adjusted in software or by adding additional diffusers/pieces paper etc. So that leaves only gain and exposure time to play with. And if gain is not an option, it's only exposure time which will likely lead to some very long exposures. Anyway, rattling here a bit. Nothing that can be overcome, it just got me into this rather fundamental question.

Yes, I am saying the same thing as Andrea. The answer is it might be affected by gain, or better, there is no reason to expect the fixed noise must be identical.  Though possibly it might be good enough for your camera sensor and your images. 

Yes, it is a bias or offset signal added to all pixels, and results in a bias frame when you use a short exposure. Yes, if you set your offset high enough you can run with that for all gains, only downside is you lose a bit of dynamic range potentially, but generally the offset is small enough that it doesn't matter. This is in fact what I do now, I am using a higher offset at higher gain, so if I drop gain I leave the offset where it is. I assume other cameras are similar. This would yes remove my reservation about the different offset, but not the potential effect of pattern noise. Not sure what your question about calibration is. Correct calibration is:

CalibratedLight = (Light - Dark) / (Flat - FlatDark)

The offset/thermal dark noise in either case is not a light signal. Flats are used to map the non-uniform light response, so if you don't subtract the respective darks from the lights and flats you will not get the correct light signal correction. Some people get away with this if there is no vignetting, but I have an SCT which cuts into my APS-C field, if I don't do the darks absolutely right (light and dark match settings, while flat and flat dark match each other) then my field is not properly corrected when I stretch. As I mentioned I find gain matching light and flats works better too. My camera is very linear response over the range I use (SharpCap has a nice routine to semi-automatically work you through testing out your camera--you can determine linearity, read noise, etc), so exposure differences for flats do not affect the linearity compared to lights. As for noise, remember each flat has the same read noise as a light. The good thing as you point out is that the flats are divided (or multiplied) depending how your SW handles them, which means the S/N is what is important. And as you say the signal is strong. However, if you do hundreds of images and only 10 flats, the noise in the flats can contribute. Note as well fixed pattern noise is not averaged in flats, so you do not want the fixed pattern noise to be different in flats and lights, it will show up (dithering your lights helps this, as it moves the target relative to the flats, averaging out the pattern noise--but better not to have it to begin with).

I use an artists A3 flat panel from Amazon with my 9.25" SCT, probably this would be good up to 10 or 11" scope, and was only about $50 if I recall. It also has 3 light settings, so is adjustable. I set it at the lowest value for high gain then adjust exposure. Uniformity from it has never been an issue. My own flats are usually a fraction of a second at high gain. At low gain I turn up the light. Even running one hundred flats is done is less than a minute. The RASC robotic 16" scope uses just sky flats, works well, though each exposure may need to be slightly different unless you can take them all very quickly one after the other.

My advice always is integrate advice, be wary of absolute NO/YES statements with no reasoning, the theory is around if you look for it, and put it all together as best you can, and then test it yourself. Note measuring noise accurately is not that easy. So for example, take a good set of lights so you noise will be low. Then use just a few flats a few darks and change things up. Stretch all images the same exactly, then see what noise and flatness look like. Come to your own conclusions of what affects what for you. Taking extra flats and darks with varying conditions is not a lot of work to understand your system--and there are many cloudy nights to experiment. It is worth it, the unfortunate fact, that is not often talked about, is the longer your total exposure and the better the image data, the better your flats and darks need to be, and the more of them you need if you really want to get all you can out of that data.
Rick

As you may have seen from my earlier post, yours and Andrea's suggestion turned out to be true for my camera. The fixed pattern noise under illumination does change with gain. It's a very small effect, and I'm not sure if it would be visible, but knowing that it's there would be enough reason to try to match them. Like you say, just taking an advice with no reasoning bears the risk that it is not applicable to your situation. That's why like you, I prefer to understand the issue and do the proper experiments.

A quick question on your artists LED flat panel. Is there a minimum exposure time you need to hit to not see any banding? I've got a Flatmaster LED panel laying here and it shows severe banding in everything under 1s. Over 1s it averages out, but 2-3s is safer. Also, is the light from the artists panel uniformly distributed?

Interesting point you bring up on # flats and darks when you have longer exposures/more frames of your lights. I see myself more often having over a 100 light frames for one filter, so perhaps calibrating it with 25 flats and 50 darks would add more noise than it reduces.

Tim Hawkes:

Willem Jan Drijfhout:
to be seen how big of an impact this has on actual images.

Interesting.  It is probably possible to work through some numbers to calculate the potential for any impact?  Trouble is questions always beget more questions and rarely answers? :-) 

Firstly am just wondering about

1)  What is the amplitude of the difference signal  relative to the  expected error  level  ( ~  200 ADU ?)   

2)  Why is there a numerically overall net positive difference signal (rather than a signal about equally populated with black zero pixels as with positive pixels) at all ?  It  seems most likely that one image actually ended up just a bit more saturated than the other (say 22.5K and 22K  thus giving a mean signal of 500 etc) -  raising the question of whether the same or some different signal would arise from any other arbitrary close to starting balance  of the two signals?

Because of the problem of  trying to measure what is likely to be a small number from the difference between two large numbers  I am thinking that this is difficult.   But also to add that --if the difference signal is down at something like 0.5% amplitude of the 22k signal -  then not only is it an awful lot of work to measure but it would also have very little potential for impact  surely  --  i.e.   a +/- 1 %  variation in some adjacent pixels being beneath notice in most images  -  and anyway lost in the effect of dithering -  where dithering is essential in any case to lose the bigger impact of pattern effects  in the dark subtraction?



Tim

That's the rabbit hole indeed, more answers > more questions....

1. Very difficult to measure, but comparing some tiny preview areas around the bands suggests something in the order of 300-400 ADU at the 'worst' places.

2. Sorry, I should have mentioned this. Since the saturation was so similar, I added a pedestal to one image before I subtracted them, just to avoid I would loose information when in certain areas the result would be smaller than 0. 

I'm with you, it is hard to imagine that the variation in pattern will have a visible effect on the final images, especially when taking things like dithering, meridian flip etc.  into account as well.

Don’t put any value into the absolute value of the difference picture. It is around 20,000 ADU, but that is because I brought it there using a pedestal, purely arbitrary. The 300 ADU is a (very rough) estimate of the maximum pattern effect. 
In comparison, a dust spot in the same flat is about a 6000 ADU difference. So one way of saying it is that the pattern effect is only 5% of the dust spot effect. Probably small enough to not be very noticeable, but large enough to not be completely discarded either.

andrea tasselli:
which makes even the tiniest difference will crop up in the final integrated light

Not if you dither though Andrea. 

And even if you didn't  dither for some reason  a PRNU -based correction in a Flat  surely can't work  because (even leaving the question of gain aside)  the correct correction depends directly  on the actual saturation of the pixel.  So the 0.2% or whatever pixel to pixel difference correction factor that would be applied by a flat captured at 20,000 ADU  wouldn't be  appropriate to apply to  either darker parts of the image  - at say 200 ADU -  where the correction would need to be much higher - say 10% - or indeed to more saturated parts where it would need to  be less.

To me it just adds complexity for no real gain (sorry about the pun) - and indeed rather than have flats at the same gain as lights many would say just use the lowest gain you can so as to get the least noise in the flats that you average together?  

So I'm still firmly in the no camp