INTRODUCTION
This post presents a calibration strategy that removes virtually all of the fixed-pattern background in most deep-sky images taken with GSENSE4040-based cameras. This chip has a so-called scientific CMOS (sCMOS) architecture, which provides dual-gain readouts of each pixel. Cameras with this chip include the Finger Lakes KL4040, the SBIG AC4040, the Moravian C4-16000, and the QHY4040PRO. Some of these cameras have been promoted as replacements to CCD cameras based on the popular Kodak KAF-16803. However, the GSENSE4040 has an unusual and potentially intrusive pattern background that can compete with good target data at low light levels, unless it is removed in post-processing.
While this fixed-pattern background (FPB) is sometimes referred to as fixed-pattern "noise" (FPN), it is an intrinsic and persistent part of the signal generated by the 4040, and unlike actual noise (readout and shot), it cannot be suppressed by taking more data. This distinctive pattern appears to be generated by the novel dual-gain readout architecture of sCMOS chips.
Moravian Instruments is upfront about the astrophotography implications of the FPB produced by the 4040, stating on the web page for their C4-16000 that "aesthetic astro-photography can be negatively influenced if these differences are not removed during image processing." Unfortunately, Moravian does not actually provide a processing strategy to remove the FPB, at least not in publicly-available documentation that I can find, and the recommendations that are currently provided by other manufacturers often fail to give adequate results. Some of the frustrations with the 4040 FPB that have been experienced by several astrophotographers are discussed at length in this Cloudy Nights thread.
I own the FLI KL4040 with a front-illuminated version of the chip, and for the past two years I've used the calibration procedure detailed in this post to remove all traces of the FPB from my images, without loss of good-quality data at low light levels. This calibration strategy differs from the standard two-step procedure of dark-subtraction and flat-field division only in how the exposures for the flat frames are chosen. The method is presented in detail further down, with illustrations using high-quality light-frame stacks for two targets.
My KL4040 images can be viewed on my Astrobin page, and I was lucky enough to have two of them appear in APOD: the Hercules Galaxy Cluster in 2020, and the Eastern Veil nebulain 2021. In addition, several owners of 4040-based cameras have asked about my approach, and were able to use it to dramatically improve the quality of their images. It seems plausible that the same strategy will also work with other dual-gain chips, such as the GSENSE2020, and the back-illuminated version of the GSENSE4040. However, I have not had access to images produced with other dual-gain chips, and can only vouch for the effectiveness of this approach with the front-illuminated GSENSE4040.
OVERVIEW OF THE CALIBRATION TECHNIQUE
I give two examples further down that illustrate the results of the proposed calibration technique when applied to real images. But it will prove useful to start with an outline of the procedure, and the reasons why something like it is often necessary.
For dark frames, one should follow the standard approach that is usually recommended for CMOS chip, which is to use exactly the same exposure time (and temperature) as the light frames, owing to the much larger dark current of most CMOS chips compared with CCDs; in other words, it is best to avoid using bias frames to interpolate the exposure times. This recommendation may be even more important for the 4040 (and possibly for other sCMOS chips), due to its intrusive FPB. Moreover, using a master bias does not remove the FPB.
On the other hand, to acquire flat frames for the 4040, it is often essential to adopt a fundamentally different criterion for the exposures compared with most CCDs and other CMOS chips. For conventional good-quality chips, the standard recommendation is to adjust the exposure time (and/or signal strength) to get a chip response at about 50% of saturation; this approach assumes a high-degree of linearity in the chip response, except near saturation, which implies that individual flat frames will differ from one another only by overall factors (except for noise, all other things being equal), and can be rescaled to a common mean value that divides out when the flat master is actually used.
Published data for the response of the 4040 does show a high-degree of linearity, but the examples below strongly suggest that the low-level FPB has a nonlinear dependence on the signal strength that is significant for deep-sky astrophotography (on the other hand, for applications at high illumination levels, the FPB will be swamped by shot noise). Consequently, for many deep-sky targets, an intrusive amount of FPB may remain in the calibrated light frames, unless the exposures for the flats are chosen to closely match their mean values to the light-frame mean; this serves to roughly equalize the strength of the FPB in the two image types. Fortunately, an exact match is not needed for adequate suppression of the FPB. Empirically, it appears that a difference of a few hundred ADUs between the flats and lights is usually acceptable. Alternatively, a good rule-of-thumb is to build the master flat from a stack of frames with means that cover roughly the same range as the stack of light frames.
To avoid introducing an excessive amount of noise from flat-frame exposures at that level, I find it necessary to stack a much larger number of flats than is usually the case with other chips. This is because the shot noise in the individual flats and lights will necessarily be similar, if their means are similar. In contrast, with conventional chips, where the flats are typically acquired at about 50% of saturation, the flat-frame noise is small relative to the light frames of most deep-sky targets. The number of flats can be minimized by applying aggressive noise reduction to the master flat; I typically apply a strong instance of PixInsight's MureDenoise script (variance set to 3.0), and then remove the first two layers. Nevertheless, I typically acquire 50-100 flat frames to beat down the noise in the master to the point where I can't notice any added noise in the calibrated light frames. I use twilight flats, and explain my strategy for acquiring them near the end of the post.
GENERAL COMMENTS
1) There are some excellent 4040 images out there, but I have not been able to find information on the post-processing strategies that were used. And there are situations where the FPB is not very intrusive and can simply be ignored, such as may happen when the target produces a much larger contribution to the chip response than the pattern background. Furthermore, when there is not enough good data at low light levels, it may be possible to suppress the FPB enough by brute force, using an aggressive black point, without compromising the rest of the image. But it often happens that the FPB competes with dim but good-quality parts of the data, and then brute force approaches to suppressing it will cause an unacceptable loss of data and image quality.
2) The PixInsight CanonBandingReduction script, developed by Georg Viehoever, can significantly reduce the FPB, as discussed in the CloudyNights post on the 4040 mentioned above. This adds some extra noise to the image, and does not suppress the FPB as effectively as an optimized flat. On the other hand, some extra effort is required to create an optimized master flat using the approach described here. Individual mileage may vary! I recommend to first try the PixInsight script if you have a 4040-based camera and have not done so already, and if so inclined, compare with the processing examples shown below, to get some sense of the relative quality of the two procedures, in the light of the additional overhead for creating optimized master flats.
3) I have seen some discussion about using dithering at extremely large amplitudes to partially suppress the 4040's FPB, but I find that the impact of aggressive dithering is limited, and in any case is unnecessary, if optimized master flats are used. I dither using amplitudes of just a few pixels, as it typical with other chips.
4) If I've missed any other useful information out there, or if anything I've written here could use correcting, or clarification, I would welcome hearing about it, so please fire away with posts here, or messages to me! I have also posted this entire screed on CloudyNights.
FPB IN DARKS, FLATS, AND LIGHTS
A vivid illustration of the FPB produced by the 4040 chip is given by this screen capture of two master darks created with my FLI 4040:
The master dark on the left is from a stack of fifty 0.1-sec exposures at -10C, while the master on the right is from a stack of thirty-five 120-sec exposures, also at -10C. All frames were acquired using the HDR ("merged") camera mode, which automatically combines the low- and high-gain 12-bit ADC outputs on the fly, to produce a (pseudo) 16-bit image. The question of which camera mode to use is addressed in detail near the end of the post. The images were in linear form and displayed with a standard screen stretch in PixInsight.
These examples show that the FPB consists of two components: a thicket of vertical bands, and seams across four quadrants. The quadrants are present because this architecture uses four adjacent chips, whose outputs are combined to produce the overall image. Reflections from the surface of the chip can reveal the quadrants, as seen for example in this webpage for the QHY4040.
The FPB in the master dark on the right is partially obscured by hot pixels, which dominate the screen stretch, given the much longer exposure time than the master on the left, and the chip's appreciable dark current. This illustrates a more general point, which is that the FPB, in a given image of any type, may, or may not, be "obvious". That depends on several factors, which include the image type, the exposure time, the nature of the target, and what kind of post-processing may have been applied. But the FPB is always present in the raw data in darks, flats, and lights.
This screen capture shows a dark-subtracted master flat on the left, and a single dark-subtracted luminance frame of M77 on the right (240-sec exposure):
The master flat is from a stack of nineteen 0.1-sec twilight-flat frames, each of which was dark-subtracted before combining into the master (the mean ADU counts of the dark-subtracted flats ranged from about 1500 to 2000). These examples illustrate the important fact that intrusive FPB is present in flats and lights even after dark subtraction, though it often happens that the FPB is not "obvious" in individual dark-subtracted light frames (without careful inspection), but comes back to bite when many light frames are stacked, if an optimized master flat is not used.
Since the FPBs in dark-subtracted flats and lights are so similar, one might expect that flat division will remove it from the light frames. However, as argued above, an intrusive amount of FPB may remain even after flat calibration, unless the flat-field exposures are chosen to closely match the means of the flats to the light-frame means. This is demonstrated in the two examples coming up.
TWO SAMPLE IMAGING TARGETS
#1: M77 LUMINANCE STACK
The screen capture below compares the results of integrating the same stack of sixty-eight dark-subtracted luminance images of M77 (240-sec exposures), but with the flat calibration done using five different master flats, except at the top-left, where no flat calibration was done. The five masters were obtained from five different flat-frame stacks with means that spanned different ADU ranges, which are listed below. The light-frame stack had means in the range 450-650 ADU (all quoted ranges are approximate). All images were in linear form, and displayed with a standard PixInsight screen stretch, and all frames were again acquired in the camera's HDR mode (the discussion of which camera mode to use is coming up).
Top row, left to right: No flat calibration; Calibration using flat-frames with means in the range 50-100 ADU; Calibration with flat-frame range 150-300.
Bottom row: Range 450-650, which roughly matches the light-frame stack; Range 750-1000; Range 2000-2400.
There is a clear trend. The FPB is very intrusive in the top-left stack, where flat-calibration was not done; it is almost completely removed in the stack on the bottom-left, where the range of flat-frame means was chosen to closely match the range of light-frame means; and the FPB is again very intrusive in the stack on the bottom-right, where the means of the flats and lights had the greatest mismatch. The other cases show a gradual improvement leading up to the "optimal" stack, and then a gradual deterioration away from it.
In case the comparisons are unclear at this small scale, here is a closeup along the centre-left edge of two cases:
Left: Optimal flat mean-ADU range 450-650; Right: flat range 2000-2400.
Four more comments to round out this example.
1) In the "optimal" flat calibration for this target, there is still a trace of the horizontal seam between the two chip quadrants on the left side of the image. That some vestiges of the FPB may remain with this approach is not surprising, since I've chosen to create the master flat from a stack of flat frames with different means, which by construction cannot exactly match the mean of every individual light frame (if any). In any event, an "exact" match between flats and lights is not possible in general, since the illumination across the flats and lights will usually be different, even if their overall means are identical. On the other hand, the visibility of the seams has been dramatically minimized using the "optimal" flat, and what little is left can be removed with some simple additional post-processing. In the case of my posted M77 image, I created a mask to select the seam that was not suppressed enough by the calibration, and used CurveTransformations to brighten that area to match the surrounding region. No trace of the FPB, including the seam, is visible in the posted image, and I did not have to resort to an aggressive black point to suppress any of it.
2) The seams between the quadrants are not usually a problem after a reasonably-optimized flat calibration, and the next example target is a more typical case, where no further correction is needed. This M77 image is a fairly extreme case in this regard, evidently because an extremely-bright nucleus is centred on the common corner between the quadrants, leading to a strong illumination of the seams. In situations like this, it is usually the case that the only trace of a seam that might be left after optimal flat calibration is the horizontal one on the left side of the image, but I don't know why the other seams don't also show up.
3) The highest flat-frame ADU range of 2000-2400 that I used here shows that the currently-available manufacturer recommendations for calibration will not work in some, if not most, cases. One manufacturer's recommendation is to use flats at 30-50% of saturation, which is about 10X higher than the highest range shown above. Moreover, at that level, the flats will come entirely from the low-gain channel (as explained below), while the light frames here come almost entirely from the high-gain output, as will usually be the case with deep-sky targets (see the section on camera modes below for more info). Another manufacturer recommends taking flats at about 1/2 of the 12-bit high-gain channel output, or about 2000 ADU, which as shown above does not produce an acceptable result.
4) Whether or not any residual FPB in a calibrated light-frame stack is a problem, including when the flats are not fully optimized, depends on the details of the target, and the aggressiveness of the rest of the post-processing, and perhaps other factors. But I think that the FPB with this camera will invariably come back to haunt the imager, if it is not dealt with effectively.
#2: ABELL 426 LUMINANCE STACK
The following screen capture compares the result of integrating the same stack of seventy-six dark-subtracted luminance exposures of Abell426 (240-sec), this time with the flat-calibration handled in three ways:
Left to right: No flat calibration; Calibration with flat means in the range 350-550, which roughly matches the light-frame range; Flat-frame range 2000-24000.
The FPB is less prominent overall with this target than in the M77 example above, evidently because the illumination of the frame is fairly uniform, with localized features that are not excessively bright, and with relatively little interesting elements in the dim regions of the frame. The flat-calibration is again optimized by roughly matching the range of ADU counts of the flats to the lights. Despite the fairly benign overall appearance of the three examples at this scale, the FPB is noticeable, except with the optimized master flat, as seen the following closeups of the image centres in two cases:
Left: Optimal flat range 350-550; Right: flat range 2000-2400.
Whether or not the FPB in the unoptimized calibration on the right is a significant problem with this FOV depends somewhat on the goals of the image processing. The FPB still remaining in this case can be removed by brute force, using a fairly aggressive black point, with a loss of data that may be acceptable. On the other hand, as illustrated on the left, a brute force chop to the data can be avoided by using a reasonable optimization of the flat-frame stack. My final Abell 426 image is posted here.
ACQUIRING TWILIGHT FLATS
One might be concerned about the feasibility of obtaining the small ADU values that are needed for optimal flat calibration with this chip, typically just a few hundred ADUs, without using a flat panel. However, I have been using twilight flats, and find that an exposure time of 0.1-sec, starting roughly 20 minutes after sunset, gives plenty of frames with suitable mean ADU counts, down to less than 50 ADU, well before drifting stars become a problem. With CCDs, such short exposures would not be feasible, due to the severe shutter shadow that would result, but the rolling-readout mode of CMOS chips allows the shutter (if any is present!) to remain open during the entire acquisition process. I've found that four or five twilight runs may be necessary to get enough flats to cover the various ADU ranges that are relevant to my images.
TO MERGE OR NOT TO MERGE?
Last topic!
GSENSE4040-based cameras allow the user to save the dual low- and high-gain 12-bit ADC outputs as separate images, as well as to have the camera merge the two channels on the fly, producing a single "HDR" (or "merged") image that is supposed to mimic the output of a genuine 16-bit ADC.
At least one manufacturer recommends to calibrate the low- and high-gain images separately, which requires saving both channel outputs for all three image types: darks, flats, and lights. In this approach, one prepares separate dark- and flat-masters for each channel, which are used to independently calibrate the corresponding light-frame stacks, with the calibrated stacks integrated separately. Calibration is completed by merging the two integrated outputs to produce a single calibrated 16-bit master light frame (details on the merging algorithm coming up). The rest of the post-processing (noise reduction, de-linearization, RGB combination, etc.) is done using conventional techniques.
This two-channel calibration strategy obviously places a significant additional burden on the imager compared with using chips with genuine 16-bit ADCs, and does not avoid the need to optimize the flats. Moreover, the software currently provided by the manufacturers does not implement this procedure for post-processing, at least not for important platforms like PixInsight. This is not an ginormous barrier to using the 4040, and I've implemented a simple version of the two-channel calibration procedure in PixInsight.
But who really wants to do all that extra work???!!!
Fortunately, I think that for the vast majority of deep-sky targets, it is not necessary to deal directly with the two channel outputs!
The claim here is that high-quality results can usually be obtained using the single HDR 16-bit output produced by the camera on the fly, which makes the entire acquisition and processing chain almost identical to what is routinely done with conventional CCDs and CMOS chips, except for the need to optimize the flat-frame exposures.
The reason is that almost all of the image data for most deep-sky targets comes from the high-gain channel. For example, with the recommended settings for the FLI KL4040, only those parts of the image above 3800 ADUs will come from the low-gain output, and that will usually occur only in small, localized regions of the target, mainly the cores of bright stars and galactic nuclei. Moreover, flats with small mean-ADU counts that are acquired in HDR mode will likewise come almost entirely from the high-gain channel. Consequently, an optimized HDR master flat will properly calibrate the high-gain parts of HDR light frames, with calibration errors arising at some level only near the brightest parts of the image. If the bright regions are small and localized, the calibration errors are not likely to be noticeable (and in any event, calibration is never perfect in practice, with any chip!). I have not noticed problems with this approach in any of my images. Although one might be inclined by this logic to use only the high-gain output, there is no advantage to doing so, and in fact, one would needlessly saturate moderately bright regions of the image, near the limit of the 12bit ADC (4096 ADU).
For the record, here is how the low- and high-gain channels are merged in HDR mode. The high-gain output is used for pixels where that channel's 12-bit ADC output lies below some threshold, otherwise the low-gain output is used by mapping it to the 16-bit ADU range above the threshold, using a multiplier and offset that are chosen in part to give continuity at the threshold. The manufacturers provide recommended values for the settings, although they can also be changed by the user. In the case of my KL4040, I use the recommended settings, with the gains of the low- and high-gain channels set to 2.8 and 16.5, respectively, the cross-over threshold set to 3800 ADU, and the multiplier and offset for the low-channel map set to 20.5 and -1228, respectively.
CONCLUSIONS
No conclusions, except to say that if anyone has the patience to read this long post, and has any constructive comments, criticisms, or additional information to add, please fire away!
This post presents a calibration strategy that removes virtually all of the fixed-pattern background in most deep-sky images taken with GSENSE4040-based cameras. This chip has a so-called scientific CMOS (sCMOS) architecture, which provides dual-gain readouts of each pixel. Cameras with this chip include the Finger Lakes KL4040, the SBIG AC4040, the Moravian C4-16000, and the QHY4040PRO. Some of these cameras have been promoted as replacements to CCD cameras based on the popular Kodak KAF-16803. However, the GSENSE4040 has an unusual and potentially intrusive pattern background that can compete with good target data at low light levels, unless it is removed in post-processing.
While this fixed-pattern background (FPB) is sometimes referred to as fixed-pattern "noise" (FPN), it is an intrinsic and persistent part of the signal generated by the 4040, and unlike actual noise (readout and shot), it cannot be suppressed by taking more data. This distinctive pattern appears to be generated by the novel dual-gain readout architecture of sCMOS chips.
Moravian Instruments is upfront about the astrophotography implications of the FPB produced by the 4040, stating on the web page for their C4-16000 that "aesthetic astro-photography can be negatively influenced if these differences are not removed during image processing." Unfortunately, Moravian does not actually provide a processing strategy to remove the FPB, at least not in publicly-available documentation that I can find, and the recommendations that are currently provided by other manufacturers often fail to give adequate results. Some of the frustrations with the 4040 FPB that have been experienced by several astrophotographers are discussed at length in this Cloudy Nights thread.
I own the FLI KL4040 with a front-illuminated version of the chip, and for the past two years I've used the calibration procedure detailed in this post to remove all traces of the FPB from my images, without loss of good-quality data at low light levels. This calibration strategy differs from the standard two-step procedure of dark-subtraction and flat-field division only in how the exposures for the flat frames are chosen. The method is presented in detail further down, with illustrations using high-quality light-frame stacks for two targets.
My KL4040 images can be viewed on my Astrobin page, and I was lucky enough to have two of them appear in APOD: the Hercules Galaxy Cluster in 2020, and the Eastern Veil nebulain 2021. In addition, several owners of 4040-based cameras have asked about my approach, and were able to use it to dramatically improve the quality of their images. It seems plausible that the same strategy will also work with other dual-gain chips, such as the GSENSE2020, and the back-illuminated version of the GSENSE4040. However, I have not had access to images produced with other dual-gain chips, and can only vouch for the effectiveness of this approach with the front-illuminated GSENSE4040.
OVERVIEW OF THE CALIBRATION TECHNIQUE
I give two examples further down that illustrate the results of the proposed calibration technique when applied to real images. But it will prove useful to start with an outline of the procedure, and the reasons why something like it is often necessary.
For dark frames, one should follow the standard approach that is usually recommended for CMOS chip, which is to use exactly the same exposure time (and temperature) as the light frames, owing to the much larger dark current of most CMOS chips compared with CCDs; in other words, it is best to avoid using bias frames to interpolate the exposure times. This recommendation may be even more important for the 4040 (and possibly for other sCMOS chips), due to its intrusive FPB. Moreover, using a master bias does not remove the FPB.
On the other hand, to acquire flat frames for the 4040, it is often essential to adopt a fundamentally different criterion for the exposures compared with most CCDs and other CMOS chips. For conventional good-quality chips, the standard recommendation is to adjust the exposure time (and/or signal strength) to get a chip response at about 50% of saturation; this approach assumes a high-degree of linearity in the chip response, except near saturation, which implies that individual flat frames will differ from one another only by overall factors (except for noise, all other things being equal), and can be rescaled to a common mean value that divides out when the flat master is actually used.
Published data for the response of the 4040 does show a high-degree of linearity, but the examples below strongly suggest that the low-level FPB has a nonlinear dependence on the signal strength that is significant for deep-sky astrophotography (on the other hand, for applications at high illumination levels, the FPB will be swamped by shot noise). Consequently, for many deep-sky targets, an intrusive amount of FPB may remain in the calibrated light frames, unless the exposures for the flats are chosen to closely match their mean values to the light-frame mean; this serves to roughly equalize the strength of the FPB in the two image types. Fortunately, an exact match is not needed for adequate suppression of the FPB. Empirically, it appears that a difference of a few hundred ADUs between the flats and lights is usually acceptable. Alternatively, a good rule-of-thumb is to build the master flat from a stack of frames with means that cover roughly the same range as the stack of light frames.
To avoid introducing an excessive amount of noise from flat-frame exposures at that level, I find it necessary to stack a much larger number of flats than is usually the case with other chips. This is because the shot noise in the individual flats and lights will necessarily be similar, if their means are similar. In contrast, with conventional chips, where the flats are typically acquired at about 50% of saturation, the flat-frame noise is small relative to the light frames of most deep-sky targets. The number of flats can be minimized by applying aggressive noise reduction to the master flat; I typically apply a strong instance of PixInsight's MureDenoise script (variance set to 3.0), and then remove the first two layers. Nevertheless, I typically acquire 50-100 flat frames to beat down the noise in the master to the point where I can't notice any added noise in the calibrated light frames. I use twilight flats, and explain my strategy for acquiring them near the end of the post.
GENERAL COMMENTS
1) There are some excellent 4040 images out there, but I have not been able to find information on the post-processing strategies that were used. And there are situations where the FPB is not very intrusive and can simply be ignored, such as may happen when the target produces a much larger contribution to the chip response than the pattern background. Furthermore, when there is not enough good data at low light levels, it may be possible to suppress the FPB enough by brute force, using an aggressive black point, without compromising the rest of the image. But it often happens that the FPB competes with dim but good-quality parts of the data, and then brute force approaches to suppressing it will cause an unacceptable loss of data and image quality.
2) The PixInsight CanonBandingReduction script, developed by Georg Viehoever, can significantly reduce the FPB, as discussed in the CloudyNights post on the 4040 mentioned above. This adds some extra noise to the image, and does not suppress the FPB as effectively as an optimized flat. On the other hand, some extra effort is required to create an optimized master flat using the approach described here. Individual mileage may vary! I recommend to first try the PixInsight script if you have a 4040-based camera and have not done so already, and if so inclined, compare with the processing examples shown below, to get some sense of the relative quality of the two procedures, in the light of the additional overhead for creating optimized master flats.
3) I have seen some discussion about using dithering at extremely large amplitudes to partially suppress the 4040's FPB, but I find that the impact of aggressive dithering is limited, and in any case is unnecessary, if optimized master flats are used. I dither using amplitudes of just a few pixels, as it typical with other chips.
4) If I've missed any other useful information out there, or if anything I've written here could use correcting, or clarification, I would welcome hearing about it, so please fire away with posts here, or messages to me! I have also posted this entire screed on CloudyNights.
FPB IN DARKS, FLATS, AND LIGHTS
A vivid illustration of the FPB produced by the 4040 chip is given by this screen capture of two master darks created with my FLI 4040:
The master dark on the left is from a stack of fifty 0.1-sec exposures at -10C, while the master on the right is from a stack of thirty-five 120-sec exposures, also at -10C. All frames were acquired using the HDR ("merged") camera mode, which automatically combines the low- and high-gain 12-bit ADC outputs on the fly, to produce a (pseudo) 16-bit image. The question of which camera mode to use is addressed in detail near the end of the post. The images were in linear form and displayed with a standard screen stretch in PixInsight.
These examples show that the FPB consists of two components: a thicket of vertical bands, and seams across four quadrants. The quadrants are present because this architecture uses four adjacent chips, whose outputs are combined to produce the overall image. Reflections from the surface of the chip can reveal the quadrants, as seen for example in this webpage for the QHY4040.
The FPB in the master dark on the right is partially obscured by hot pixels, which dominate the screen stretch, given the much longer exposure time than the master on the left, and the chip's appreciable dark current. This illustrates a more general point, which is that the FPB, in a given image of any type, may, or may not, be "obvious". That depends on several factors, which include the image type, the exposure time, the nature of the target, and what kind of post-processing may have been applied. But the FPB is always present in the raw data in darks, flats, and lights.
This screen capture shows a dark-subtracted master flat on the left, and a single dark-subtracted luminance frame of M77 on the right (240-sec exposure):
The master flat is from a stack of nineteen 0.1-sec twilight-flat frames, each of which was dark-subtracted before combining into the master (the mean ADU counts of the dark-subtracted flats ranged from about 1500 to 2000). These examples illustrate the important fact that intrusive FPB is present in flats and lights even after dark subtraction, though it often happens that the FPB is not "obvious" in individual dark-subtracted light frames (without careful inspection), but comes back to bite when many light frames are stacked, if an optimized master flat is not used.
Since the FPBs in dark-subtracted flats and lights are so similar, one might expect that flat division will remove it from the light frames. However, as argued above, an intrusive amount of FPB may remain even after flat calibration, unless the flat-field exposures are chosen to closely match the means of the flats to the light-frame means. This is demonstrated in the two examples coming up.
TWO SAMPLE IMAGING TARGETS
#1: M77 LUMINANCE STACK
The screen capture below compares the results of integrating the same stack of sixty-eight dark-subtracted luminance images of M77 (240-sec exposures), but with the flat calibration done using five different master flats, except at the top-left, where no flat calibration was done. The five masters were obtained from five different flat-frame stacks with means that spanned different ADU ranges, which are listed below. The light-frame stack had means in the range 450-650 ADU (all quoted ranges are approximate). All images were in linear form, and displayed with a standard PixInsight screen stretch, and all frames were again acquired in the camera's HDR mode (the discussion of which camera mode to use is coming up).
Top row, left to right: No flat calibration; Calibration using flat-frames with means in the range 50-100 ADU; Calibration with flat-frame range 150-300.
Bottom row: Range 450-650, which roughly matches the light-frame stack; Range 750-1000; Range 2000-2400.
There is a clear trend. The FPB is very intrusive in the top-left stack, where flat-calibration was not done; it is almost completely removed in the stack on the bottom-left, where the range of flat-frame means was chosen to closely match the range of light-frame means; and the FPB is again very intrusive in the stack on the bottom-right, where the means of the flats and lights had the greatest mismatch. The other cases show a gradual improvement leading up to the "optimal" stack, and then a gradual deterioration away from it.
In case the comparisons are unclear at this small scale, here is a closeup along the centre-left edge of two cases:
Left: Optimal flat mean-ADU range 450-650; Right: flat range 2000-2400.
Four more comments to round out this example.
1) In the "optimal" flat calibration for this target, there is still a trace of the horizontal seam between the two chip quadrants on the left side of the image. That some vestiges of the FPB may remain with this approach is not surprising, since I've chosen to create the master flat from a stack of flat frames with different means, which by construction cannot exactly match the mean of every individual light frame (if any). In any event, an "exact" match between flats and lights is not possible in general, since the illumination across the flats and lights will usually be different, even if their overall means are identical. On the other hand, the visibility of the seams has been dramatically minimized using the "optimal" flat, and what little is left can be removed with some simple additional post-processing. In the case of my posted M77 image, I created a mask to select the seam that was not suppressed enough by the calibration, and used CurveTransformations to brighten that area to match the surrounding region. No trace of the FPB, including the seam, is visible in the posted image, and I did not have to resort to an aggressive black point to suppress any of it.
2) The seams between the quadrants are not usually a problem after a reasonably-optimized flat calibration, and the next example target is a more typical case, where no further correction is needed. This M77 image is a fairly extreme case in this regard, evidently because an extremely-bright nucleus is centred on the common corner between the quadrants, leading to a strong illumination of the seams. In situations like this, it is usually the case that the only trace of a seam that might be left after optimal flat calibration is the horizontal one on the left side of the image, but I don't know why the other seams don't also show up.
3) The highest flat-frame ADU range of 2000-2400 that I used here shows that the currently-available manufacturer recommendations for calibration will not work in some, if not most, cases. One manufacturer's recommendation is to use flats at 30-50% of saturation, which is about 10X higher than the highest range shown above. Moreover, at that level, the flats will come entirely from the low-gain channel (as explained below), while the light frames here come almost entirely from the high-gain output, as will usually be the case with deep-sky targets (see the section on camera modes below for more info). Another manufacturer recommends taking flats at about 1/2 of the 12-bit high-gain channel output, or about 2000 ADU, which as shown above does not produce an acceptable result.
4) Whether or not any residual FPB in a calibrated light-frame stack is a problem, including when the flats are not fully optimized, depends on the details of the target, and the aggressiveness of the rest of the post-processing, and perhaps other factors. But I think that the FPB with this camera will invariably come back to haunt the imager, if it is not dealt with effectively.
#2: ABELL 426 LUMINANCE STACK
The following screen capture compares the result of integrating the same stack of seventy-six dark-subtracted luminance exposures of Abell426 (240-sec), this time with the flat-calibration handled in three ways:
Left to right: No flat calibration; Calibration with flat means in the range 350-550, which roughly matches the light-frame range; Flat-frame range 2000-24000.
The FPB is less prominent overall with this target than in the M77 example above, evidently because the illumination of the frame is fairly uniform, with localized features that are not excessively bright, and with relatively little interesting elements in the dim regions of the frame. The flat-calibration is again optimized by roughly matching the range of ADU counts of the flats to the lights. Despite the fairly benign overall appearance of the three examples at this scale, the FPB is noticeable, except with the optimized master flat, as seen the following closeups of the image centres in two cases:
Left: Optimal flat range 350-550; Right: flat range 2000-2400.
Whether or not the FPB in the unoptimized calibration on the right is a significant problem with this FOV depends somewhat on the goals of the image processing. The FPB still remaining in this case can be removed by brute force, using a fairly aggressive black point, with a loss of data that may be acceptable. On the other hand, as illustrated on the left, a brute force chop to the data can be avoided by using a reasonable optimization of the flat-frame stack. My final Abell 426 image is posted here.
ACQUIRING TWILIGHT FLATS
One might be concerned about the feasibility of obtaining the small ADU values that are needed for optimal flat calibration with this chip, typically just a few hundred ADUs, without using a flat panel. However, I have been using twilight flats, and find that an exposure time of 0.1-sec, starting roughly 20 minutes after sunset, gives plenty of frames with suitable mean ADU counts, down to less than 50 ADU, well before drifting stars become a problem. With CCDs, such short exposures would not be feasible, due to the severe shutter shadow that would result, but the rolling-readout mode of CMOS chips allows the shutter (if any is present!) to remain open during the entire acquisition process. I've found that four or five twilight runs may be necessary to get enough flats to cover the various ADU ranges that are relevant to my images.
TO MERGE OR NOT TO MERGE?
Last topic!
GSENSE4040-based cameras allow the user to save the dual low- and high-gain 12-bit ADC outputs as separate images, as well as to have the camera merge the two channels on the fly, producing a single "HDR" (or "merged") image that is supposed to mimic the output of a genuine 16-bit ADC.
At least one manufacturer recommends to calibrate the low- and high-gain images separately, which requires saving both channel outputs for all three image types: darks, flats, and lights. In this approach, one prepares separate dark- and flat-masters for each channel, which are used to independently calibrate the corresponding light-frame stacks, with the calibrated stacks integrated separately. Calibration is completed by merging the two integrated outputs to produce a single calibrated 16-bit master light frame (details on the merging algorithm coming up). The rest of the post-processing (noise reduction, de-linearization, RGB combination, etc.) is done using conventional techniques.
This two-channel calibration strategy obviously places a significant additional burden on the imager compared with using chips with genuine 16-bit ADCs, and does not avoid the need to optimize the flats. Moreover, the software currently provided by the manufacturers does not implement this procedure for post-processing, at least not for important platforms like PixInsight. This is not an ginormous barrier to using the 4040, and I've implemented a simple version of the two-channel calibration procedure in PixInsight.
But who really wants to do all that extra work???!!!
Fortunately, I think that for the vast majority of deep-sky targets, it is not necessary to deal directly with the two channel outputs!
The claim here is that high-quality results can usually be obtained using the single HDR 16-bit output produced by the camera on the fly, which makes the entire acquisition and processing chain almost identical to what is routinely done with conventional CCDs and CMOS chips, except for the need to optimize the flat-frame exposures.
The reason is that almost all of the image data for most deep-sky targets comes from the high-gain channel. For example, with the recommended settings for the FLI KL4040, only those parts of the image above 3800 ADUs will come from the low-gain output, and that will usually occur only in small, localized regions of the target, mainly the cores of bright stars and galactic nuclei. Moreover, flats with small mean-ADU counts that are acquired in HDR mode will likewise come almost entirely from the high-gain channel. Consequently, an optimized HDR master flat will properly calibrate the high-gain parts of HDR light frames, with calibration errors arising at some level only near the brightest parts of the image. If the bright regions are small and localized, the calibration errors are not likely to be noticeable (and in any event, calibration is never perfect in practice, with any chip!). I have not noticed problems with this approach in any of my images. Although one might be inclined by this logic to use only the high-gain output, there is no advantage to doing so, and in fact, one would needlessly saturate moderately bright regions of the image, near the limit of the 12bit ADC (4096 ADU).
For the record, here is how the low- and high-gain channels are merged in HDR mode. The high-gain output is used for pixels where that channel's 12-bit ADC output lies below some threshold, otherwise the low-gain output is used by mapping it to the 16-bit ADU range above the threshold, using a multiplier and offset that are chosen in part to give continuity at the threshold. The manufacturers provide recommended values for the settings, although they can also be changed by the user. In the case of my KL4040, I use the recommended settings, with the gains of the low- and high-gain channels set to 2.8 and 16.5, respectively, the cross-over threshold set to 3800 ADU, and the multiplier and offset for the low-channel map set to 20.5 and -1228, respectively.
CONCLUSIONS
No conclusions, except to say that if anyone has the patience to read this long post, and has any constructive comments, criticisms, or additional information to add, please fire away!
