Dark frames temperature and final picture quality - CMOS camera

First, I'd recommend watching Robin Glover's excellent talk.  Some answers

1. Yes, you can get hot pixels below 0C or really at any temperature
2. Particularly on the 294MM, the answer is yes, you really do want to match the light/dark temperature. The 294MM has rather prominent amp glow and using dark frames that are lower in temperature will undercorrect it, while ones higher in temperature will over correct. Either result will be visible in the calibrated frames. I've actually tried this out of curiosity.
3. -5C is high enough of a temperature that it could, theoretically, make a difference versus -15C if you're imaging from very dark skies or taking long narrow band exposures. Whether the difference is practically meaningful is impossible to tell in the absence of a lot of other information that it probably isn't very productive to attempt to get. In general, you want to run at the coldest temperature that your camera will consistently maintain and that will be different depending on the time of year. Dark libraries are not hard to build. I have dark libraries at -10, -15, -20, and -25C which covers pretty much any time of the year I want to image.

Michael Finan:
I think you my be confused about how you set the temperature. The temperature set is the desired temperature of the chip. So set -5c and the chip will cool to -5c, not 5 below ambient. I always use -10c for all my imaging. I can always get to -10c, and I often image in summertime Florida. In the summer it runs at 70-80%, on a cold (5c) night it runs at less than 30. I have no issues with using darks for several years all at -10c.

Oh my... pardon my ignorance. I always thought it was a cooling compared to ambient temperature, not the actual shooting temperature. That changes a lot of things. Maybe I read in the manual something about below ambient... not sure how I had that stuck in my head. Thanks a lot for clarifying!

When you calibrate it may cause dark spots due to overcorrecting.

That's what I was curious to know what it'd look like with different temperatures. Thank you!

In general, you want to run at the coldest temperature that your camera will consistently maintain and that will be different depending on the time of year. Dark libraries are not hard to build. I have dark libraries at -10, -15, -20, and -25C which covers pretty much any time of the year I want to image.

Alright then, better not be lazy and shoot darks during a cloudy night I guess How many would you use for a solid base / without overdoing it, from your personal experience? I was reading minimum 15, but 50 to be pretty much noise free.


Thanks all for your quick answers! Really appreciate it.

Roger Nichol:

Alright then, better not be lazy and shoot darks during a cloudy night I guess How many would you use for a solid base / without overdoing it, from your personal experience? I was reading minimum 15, but 50 to be pretty much noise free.

I recommend you shoot at least as many dark frames (per temperature / gain combinations that you use) as you will have lights. For me that means a total integration of darks to be 20+ hours as I often integrate lights to that level. If you have significantly fewer darks than lights then the maths tells us that your darks will contribute noise into the final result, as the noise remaining in the darks is correlated across each light. Dithering does mitigate this to some degree.  Since darks cost me nothing but time, I set mine up to run for a few days twice a year to capture a fresh batch at -20C for winter and -15C for summer.  On my most recent batch I used 84 x 1200s, 100 x 600s, 200 x 300s, 300 x 30s (for stars). Maybe overkill but it works for me.

If I may ask, can you point us to a book/paper as a reference? Thank you!

Björn Arnold:

Roger Nichol:

Alright then, better not be lazy and shoot darks during a cloudy night I guess How many would you use for a solid base / without overdoing it, from your personal experience? I was reading minimum 15, but 50 to be pretty much noise free.

I recommend you shoot at least as many dark frames (per temperature / gain combinations that you use) as you will have lights. For me that means a total integration of darks to be 20+ hours as I often integrate lights to that level. If you have significantly fewer darks than lights then the maths tells us that your darks will contribute noise into the final result, as the noise remaining in the darks is correlated across each light. Dithering does mitigate this to some degree.  Since darks cost me nothing but time, I set mine up to run for a few days twice a year to capture a fresh batch at -20C for winter and -15C for summer.  On my most recent batch I used 84 x 1200s, 100 x 600s, 200 x 300s, 300 x 30s (for stars). Maybe overkill but it works for me.

The noise contribution when calibrating a light frame is approximately proportional to the square root of 1/m, where m is the number of dark frames to create a master dark.

Therefore, you can use a rather low two-digit number of darks. I‘m typically using 30 darks which leads to an increase of noise of about 1,6%. 

Björn

I believe this is a situation where my favorite sentence applies: "In theory theory and practice are the same, in practice they are not."  (unknown source)

In theory Roger is right because - as noted by Björn - the higher the number of images the lower the random noise irrespective whether you take lights, darks, flats, bias, dark flats or whatever you choose otherwise.

In practice it may become irrelevant at some point. Usually effect of light pollution will be several orders of magnitude higher than the difference in noise between an average of 30 darks and an average of 1000 darks. There is nothing wrong with either approach and "the more the better" but at some point you just get diminishing returns.

In addition I assume this is where the "practice vs theoriy" part kicks in. The mathematics hold true only in the absence of systematic errors. Any practical setup will introduce some systematic errors which will set a lower limit to the level of noise reduction you can achieve.

Clear Skies
Wolfgang

The LP/Sky Background is the great leveller, so don't fuss about it. I can barely notice the difference between 10 darks and 20, when I bother to taken them 20.

Die Launische Diva:

Roger Nichol:

Alright then, better not be lazy and shoot darks during a cloudy night I guess How many would you use for a solid base / without overdoing it, from your personal experience? I was reading minimum 15, but 50 to be pretty much noise free.

I recommend you shoot at least as many dark frames (per temperature / gain combinations that you use) as you will have lights. For me that means a total integration of darks to be 20+ hours as I often integrate lights to that level. If you have significantly fewer darks than lights then the maths tells us that your darks will contribute noise into the final result, as the noise remaining in the darks is correlated across each light. Dithering does mitigate this to some degree.  Since darks cost me nothing but time, I set mine up to run for a few days twice a year to capture a fresh batch at -20C for winter and -15C for summer.  On my most recent batch I used 84 x 1200s, 100 x 600s, 200 x 300s, 300 x 30s (for stars). Maybe overkill but it works for me.

If I may ask, can you point us to a book/paper as a reference? Thank you!

I could not find a definitive mathematical paper on this so used my own analysis below:  Conclusion, if you dither sufficiently then the number of darks in your stack is not so critical - my approach is thus over-kill.


Dark frames contain bias, residual pattern interference (hot/dark pixels, 'amp noise' and other heat related artefacts plus any other fixed sensor variations) and random noise.

Integrating a stack of frames maintains the same fixed pattern (signal) and averages the random noise such that the signal-to-noise ratio improves at a rate of the square root of the number of frames. E.g. the signal to noise ratio of a stack of 64 images will be improved by a factor of 8.The integrated stack of dark frames will thus have the fixed pattern plus a reduced amount of noise - let's call that N(d).

When calibrating your lights you are (simplistically) subtracting the darks from each light. The calibrated light should ideally then have no remaining fixed sensor pattern and will have the small remaining noise N(d) superimposed on that sub's random noise.When stacking a number of calibrated subs (ignoring dithering and rejection algorithms for now) the wanted 'fixed' pattern from the target is maintained (or averaged where it is varying due to atmosphere, etc.) and the random noise is reduced at the square-root rate, let's call this N(l).  Each calibrated light also contains the identical residual master dark noise N(d) so this is maintained in the integrated stack.

If we have the same number of light and dark frames, N(d) and N(l) would be at a comparable level and neither would predominate in the ensuing processing.

If we took an example where there are 144 lights and 16 darks used, then the noise contributed by the darks would be 3x the level of noise remaining from the light stack (12/4) and would predominate in further processing.

This is, of course, an over-simplified analysis, since in practise we use dithering, cosmetic correction and rejection algorithms to remove outliers.  Of these, only dithering directly impacts the master dark residual noise N(d). If dithering is sufficiently spread, e.g. over 12+ pixels, then the N(d) noise is spread around during alignment and averaged during stacking.  If there is sufficient dithering, e.g. every frame or frequent frames where there are many frames, then this should mitigate the noise introduced from having fewer darks than lights and the residual noise in the lights stack would predominate.

By curiosity, why shoot dark libraries every year/a few times a year? Just in case the sensor gets damaged or lose its efficiency with age?



Regarding flat frames, would you follow the same logic: as many flat as light shots taken that night?

I'm assuming no one really use a library for these, due to dust moving around.

I shoot my dark libraries a couple of times each year (winter set and summer set) and do find that the number of hot pixels does increase over time, e.g. a set that I took 10 months after my first set at the same settings had an additional handful of bring hot pixels. I'm not sure whether that trend continues after the first year or so but it's not a big deal to re-do them.

For flats, I just take 50 flats and 50 dark flats for each filter. I could probably get away with less but they are quick.  I do tend to re-use my flats for a while as I keep my imaging train intact and use a refractor which is sealed.  I usually redo the flats if I rotate the camera, but I can't see any significant change in flats as I rotate - there are no visible dust artefacts from the scope and its vignetting seems symmetrical.  This probably wouldn't apply with a reflector that had a spider on its front end.

As has been pointed out, dark frames eliminate fixed pattern noise, but the notion that as many dark frames are need as light frames seems  extreme. If you are only interested in removing hot pixels,  and you are willing to dither, you only need one  dark frame at the start or at the end of a long sequence.  This save an immense amount of time. 

I got the idea for this method from reading the documentation of IRIS. In this method you use one dark frame to create a map of hot pixels. I set the iso such that the  stdev of the dark frame =1, and define any pixel larger than 10 *stdev  as "hot".  Typically, on my Z6 at iso100  and 60 seconds exposure (at about 20C)  there are about 5000 hot pixels (in 24 Mp) . These are written to a "hot pixel list". 

This is how it works out with my Nikkor 180mm ED lens (plate scale is 6"/pixel) for a sequence of 100x60sec shots.  I intentionally throw the polar alignment off , such that I get about a 1/2 pixel trail per exposure. This is a simple method of dithering.   I then use a preprocessor to read all 100 images and replace the pixels in the "hot_pixel_list" with the median of the eight surrounding pixels, then register the "cleaned" images.   The brightest  hot pixels are now 20 DN (since I use 50% dithering). The star signal has increased by a factor 100. If I were read-noise limited, the noise would have grown from 1DN to 10 DN. The 20 DN hot pixel trails  would then be  easy to see as 5000 faint 300" long trails , but in reality the trails are drowned out by the background noise, which from my backyard is a factor of 5 bigger than the read-noise. 

Sequator may be using  a similar method, since it produces good results with a single dark frame. 

Cheers.

hha

And never mind if you have amp glow or IR glow. Good luck removing them with a hot pixel list.

John Hayes:
In my case, I use 20 minute exposures so I can create a 20 frame master dark using "only" a total of 400 minutes (6h 40m) compared to taking 6,400 minutes, which is over 106 hours of time--for each channel! And...yes, I have stacked that much data for some really faint images. I personally use 17 subs for all of my master darks and master flats and it works quite well. For most of us doing long exposure imaging, using master calibration files with 12-20 subs will work just fine.

Thank you for this post, that's super interesting and graphs help a lot to visualize the data.

For 20 minutes exposures, I see how having a big dark frame library is "overkilled" and not worth the time of going with the 1 light  = 1 dark method for the noise reduction.

For lower exposures however, like 3-10 minutes, why not spend the time for 30-60 dark shots to be on the lower end of the graph? Especially if they aren't many light shots to stack. If it's only a couple hours in a day, it's not really time consuming and wouldn't hurt. Unless... we're talking gear use and damaging the equipment faster than it should... then yes, it would hurt

Do you have similar data graphs that compare the noise level with different sensor temperatures? I read that it approximately doubles every 5°C, but I'd be interested to see if you have some data gathered for that as well.

John Hayes:
In my case, I use 20 minute exposures so I can create a 20 frame master dark using "only" a total of 400 minutes (6h 40m) compared to taking 6,400 minutes, which is over 106 hours of time--for each channel! And...yes, I have stacked that much data for some really faint images. I personally use 17 subs for all of my master darks and master flats and it works quite well. For most of us doing long exposure imaging, using master calibration files with 12-20 subs will work just fine.

Thank you for this post, that's super interesting and graphs help a lot to visualize the data.

For 20 minutes exposures, I see how having a big dark frame library is "overkilled" and not worth the time of going with the 1 light  = 1 dark method for the noise reduction.

For lower exposures however, like 3-10 minutes, why not spend the time for 30-60 dark shots to be on the lower end of the graph? Especially if they aren't many light shots to stack. If it's only a couple hours in a day, it's not really time consuming and wouldn't hurt. Unless... we're talking gear use and damaging the equipment faster than it should... then yes, it would hurt

Do you have similar data graphs that compare the noise level with different sensor temperatures? I read that it approximately doubles every 5°C, but I'd be interested to see if you have some data gathered for that as well.

Perhaps you missed the point that the calculation shows the noise contribution relative to the noise in a single dark frame.  Lowering the temperature lowers the noise contribution and visa-versa but that doesn't have anything to do with the comparison of what happens as you vary the number of calibration frames that go into the master dark for any given stack.  Everything is relative to the dark noise at any given temperature.  As I said, using more data always makes things better but past a certain point, you won't see any improvement.  For example, you could, in principal, use a master dark with say 10,000 frames; but, that's a waste of time and effort.  That won't produce a result for that's noticeably better than using say 16-20 frames for most data stacks.  If the number of light frames goes down, the difference becomes even smaller.  In the limit of an infinite number of calibration frame, the noise just becomes the square root of 1/N, which is the minimum you can get.  You can use as many  frames as you like, but past a certain point, that won't make your images any better.   Try the the comparison that I suggested and I think that you will see what I am talking about.   (I'm traveling in S. America right now and I don't have time to prepare it or I'd do it for you.)

John


PS.  To avoid confusion, I want to gently point out that dark libraries contain master darks for a wide range of exposures.   Perhaps you meant "master dark" rather than "library."