Dithering & Darks

Good Day to All!

I live in Providence, RI under Bortle 8 skies

My Equipment:
Meade Series 6000 80mm Triplet APO
Sky Watcher EQ6-R Pro
Canon EOS Ra
ZWO 30mm f4 MiniScope - Guiding
ZWO ASI224mc - Guiding

Notes: I guide usingan ST-4 connection, I use PHD2 for guiding (multi-star) and I use APT for capturing
I dither 20 pixels - 4 in PHD2 and 5 in APT

My main question is do I need to take darks - I have the similar issue as someone else who I saw post a comment saying that with the DSLR obviously I cannot control my camera's sensor temperature. I have processed images in the past - stacking one set with darks and one without and it didn't seem to make much of a difference (then again, it may be my crappy processing skills)

Because I cannot control the temp, I usually take my darks right before it gets dark enough to start my polar alignment - if I am going to use my typical 3 - 4 min exposures I will begin early enough to get that done - Ambient temperature difference between then and the end of my sessions could be up to 4-5 degrees (going from 38 degrees F to 28 degrees F) Celsius depending on how long I go - weeknights I am out for around 4-5 hours, weekends could be an all nighter.

Another Note: I typically take 15-20 darks, 50 flats, 50 dark flats and 50 bias frames. My light subs are 180s if I am not using an L-eNhanced filter, and either 4 - 5 min (typically 4) if I am. My ISO is set to 800 (tempted to try 1600)

I suppose my question is this, since I am dithering sort of aggressively, and since I am using a DSLR, should I even bother taking flats? OR at the very least, maybe take one set of flats for each exposure / ISO set maybe one time a month unless the ambient temperature changes drastically.

I don't get a lot of time, like most of us, to shoot actual data so I am trying to make my data capturing, as well as my processing, as efficient as possible.

Also, if anyone has any other suggestions, please feel free - First though, yes, I know, I should connect to my computer and guide that way, yes, I know, I should connect to my computer and plate solve and do my polar alignment through sharpcap, and yes, I know I really should take the ASIair Pro I bought out of the box and use that. I am currently happy with how my set up is the moment!

Thanks to everyone in advance!

Jared -

I own and use a 5D Mark IV, same sensor as your EOS Ra. The main reason to take darks with a uncooled camera is to do a bias correction. The bias in these sensors/cameras depends on exposure time, so a bias taken at 1/8000s won't correlate in the bias of your longer exposure.

As to flats, calibrate them with darks of the same ISO/exposure length. Apart from correcting for vignetting, the reason to take flats is to correct for dust motes, the most problematic ones being the ones on your sensor or on the back of the rear element. These can move session to session, making flats from previous sessions unusable. If you don't have dust/smudges in your setup, and don't change out your lenses very much or at all, you may very possibly get away with reusing flats.

Arun

Jared -

I own and use a 5D Mark IV, same sensor as your EOS Ra. The main reason to take darks with a uncooled camera is to do a bias correction. The bias in these sensors/cameras depends on exposure time, so a bias taken at 1/8000s won't correlate in the bias of your longer exposure.

As to flats, calibrate them with darks of the same ISO/exposure length. Apart from correcting for vignetting, the reason to take flats is to correct for dust motes, the most problematic ones being the ones on your sensor or on the back of the rear element. These can move session to session, making flats from previous sessions unusable. If you don't have dust/smudges in your setup, and don't change out your lenses very much or at all, you may very possibly get away with reusing flats.

Arun

So, should I still take darks and biases then? If so, does it matter the ambient temperature (can I start even earlier or do them at different times, etc). I do take my biases at 1/8000s, should I change this or stay the same?

I take new flats and dark flats every time I image. My flats are done with a light panel and a doubled handkerchief and I have the light set so it exposes at 1.6s (same for my dark flats).

So, should I still take darks and biases then? If so, does it matter the ambient temperature (can I start even earlier or do them at different times, etc).

No, what I am suggesting is to not take biases at all. Your 1/8000s bias will not properly calibrate a sub that is several seconds or minutes long. Remember that your dark already contains the bias signal. So I would recommend the following (which is my workflow):

1. Take a set of dark flats; construct a master dark flat
2. Take your flat frames (same exposure time as the dark flat)
3. Calibrate your flats with your master dark flat - no bias needed here; this gives you your Master Flat.
4. Take a set of darks; do this close to your imaging session, either immediately before or immediately after. This makes sure they are close to your imaging temperature. What you don't want to do is take your darks indoors where it is 75 degrees, then image at 38 degrees F. If you do this, you risk adding noise when you calibrate. Your darks should be the same exposure time as lights. Construct a Master Dark. Once again, do not bother with a bias.
5. Calibrate your light frames with your previously constructed Master Flat and Master Dark.
6. Stack as normal

As Arun states, if you have Darks with the same exposure time, Bias frames should not be needed. If your darks are of a significantly different exposure duration then bias frames can be used to scale the darks, but this is not a preferred solution.

To know whether you need to use darks, stretch an integrated stack of darks (e.g. 20+) at the exposure length that you take and see what residual pattern is left. On an uncooled DSLR you will undoubtedly see all kinds of nastiness that would appear in your stacked images if you don't calibrate them out with darks.

If your darks are taken within 5C of the average temp from your sensor then they should be reasonably good. Obviously the more controlled the temp the better.

No, what I am suggesting is to not take biases at all. Your 1/8000s bias will not properly calibrate a sub that is several seconds or minutes long. Remember that your dark already contains the bias signal. So I would recommend the following (which is my workflow):

Roger Nichol:
If your darks are taken within 5C of the average temp from your sensor then they should be reasonably good. Obviously the more controlled the temp the better.

Thanks to both of you! So essentially, no bias, just darks and flats / dark flats and as long as I am outside and takign right before I begin so I don't have a huge temperature difference, I should be okay.

Is there a down side to taking bias frames - should I be restacking my previous images with no bias?

So, should I still take darks and biases then? If so, does it matter the ambient temperature (can I start even earlier or do them at different times, etc).

No, what I am suggesting is to not take biases at all. Your 1/8000s bias will not properly calibrate a sub that is several seconds or minutes long. Remember that your dark already contains the bias signal. So I would recommend the following (which is my workflow):

1. Take a set of dark flats; construct a master dark flat
2. Take your flat frames (same exposure time as the dark flat)
3. Calibrate your flats with your master dark flat - no bias needed here; this gives you your Master Flat.
4. Take a set of darks; do this close to your imaging session, either immediately before or immediately after. This makes sure they are close to your imaging temperature. What you don't want to do is take your darks indoors where it is 75 degrees, then image at 38 degrees F. If you do this, you risk adding noise when you calibrate. Your darks should be the same exposure time as lights. Construct a Master Dark. Once again, do not bother with a bias.
5. Calibrate your light frames with your previously constructed Master Flat and Master Dark.
6. Stack as normal

This is interesting. I've been using biases and no darks on the D5300 (Dithering in RA only) and getting good results (admittedly have not yet tried this in summer)

I took a 180s dark frame, then subtracted out a bias frame and the result was ... barely anything, so it seemed reasonable.

I haven't been doing dark flats but I have increased exposure on flats to 3 sec so I may play with that

I cannot leave my gear unattended, so taking a bunch of 180s darks would really need to give me a huge benefit as it eats into so much imaging time. For now, dithering really seems to close the gap and let me collect more light frames

Nick Ambrose:
This is interesting. I've been using biases and no darks on the D5300 (Dithering in RA only) and getting good results (admittedly have not yet tried this in summer)

I took a 180s dark frame, then subtracted out a bias frame and the result was ... barely anything, so it seemed reasonable.

I haven't been doing dark flats but I have increased exposure on flats to 3 sec so I may play with that

I cannot leave my gear unattended, so taking a bunch of 180s darks would really need to give me a huge benefit as it eats into so much imaging time. For now, dithering really seems to close the gap and let me collect more light frames

Yeah - I try to take my darks right before I polar align - so from around 4:00pm to 5:00pm local time (currently) - which typically leaves me within 4-5c or 6-10 F ambient temperature wise - and since its an uncooled camera I figure that is the best I'm going to get without taking a light frame then a dark frame, etc.

My flats are around 1`.6s which I like - I take dark flats but I have read if your flats match your lights in ISO / Gain you don't need to take dark flats - but I do anyways.

There are too many theories and "right" answers so I figure I will take a good data set, restack with and with out darks, with and without biases, etc and see how it goes.

The intricacies of CMOS sensors, in particular the BSI sensor used in the ASI 294MC, are covered in detail in this excellent writeup by John Upton. The recommendation to not use biases but to use darks of the same exposure duration as the frame you are trying to calibrate is derived from this technical analysis. You will also see that he recommends not using any frames less than 3s in duration for calibration. For this reason, I use flats of 4s and calibrate them with 4 s darks. I do this with all CMOS cameras I use - whether DSLR or cooled:

https://www.cloudynights.com/topic/636301-asi294mc-calibration-–-testing-notes-thoughts-and-opinions/

This probably applies to any Sony BSI sensor. Obviously, the sensors in Canon DSLRs and mirrorless are not manufactured by Sony but by Canon themselves, yet the guidelines are rational and there is probably at least some similarity between Sony and Canon sensors. For example, the dependence of bias on exposure length also appears to be true of the Toshiba sensor used in the ASI 1600MM (also CMOS, but not BSI) as detailed by Jon Rista in the same thread.

Ultimately, if your images look good to you, there is no reason to change what you are doing.

The intricacies of CMOS sensors, in particular the BSI sensor used in the ASI 294MC, are covered in detail in this excellent writeup by John Upton. The recommendation to not use biases but to use darks of the same exposure duration as the frame you are trying to calibrate is derived from this technical analysis. You will also see that he recommends not using any frames less than 3s in duration for calibration. For this reason, I use flats of 4s and calibrate them with 4 s darks. I do this with all CMOS cameras I use - whether DSLR or cooled:

https://www.cloudynights.com/topic/636301-asi294mc-calibration-–-testing-notes-thoughts-and-opinions/

This probably applies to any Sony BSI sensor. Obviously, the sensors in Canon DSLRs and mirrorless are not manufactured by Sony but by Canon themselves, yet the guidelines are rational and there is probably at least some similarity between Sony and Canon sensors. For example, the dependence of bias on exposure length also appears to be true of the Toshiba sensor used in the ASI 1600MM (also CMOS, but not BSI) as detailed by Jon Rista in the same thread.

Ultimately, if your images look good to you, there is no reason to change what you are doing.

I believe you are correct, etc! I don't dispute anything that has been said. I just know I could also ask a different group and they could say something different. I have had images that have processed better without darks, but I have only tried it like once or twice. Another forum post here, I am on my phone so I don't have the link, was saying since you cannot control the temp and its going to have a massive difference then your lights, to not do darks with a DSLR.

For DSLRs in particular, it is worth thinking about what really fast shutter speeds mean and how they are accomplished and what this means in the context of what we think are bias or "0 second exposure" frames.

DSLRs have two mechanical shutters - a first and second curtain. Exposure starts when the first shutter begins to clear the sensor. It ends when the second shutter completely blocks the sensor. Shutter speeds faster than the flash sync speed (1/180s, 1/200s or 1/250s depending on camera and sensor size) are accomplished by having the second curtain begin to cross the sensor before the first curtain has completely cleared it, i.e., only a portion of the sensor gets exposed to light at any given time.

In the context of taking darks or biases for astrophotography, however, the positions of the first and second shutters are irrelevant since no light is being recorded by the sensor anyway. What matters is how long the sensor is active, and this is probably close to or even greater than the flash sync speed. So your 1/8000s bias is really closer to a 1/200s or longer dark. This is probably another reason to avoid bias frames altogether and only use exposures for calibration that are significant multiples of the flash sync speed.

The intricacies of CMOS sensors, in particular the BSI sensor used in the ASI 294MC, are covered in detail in this excellent writeup by John Upton. The recommendation to not use biases but to use darks of the same exposure duration as the frame you are trying to calibrate is derived from this technical analysis. You will also see that he recommends not using any frames less than 3s in duration for calibration. For this reason, I use flats of 4s and calibrate them with 4 s darks. I do this with all CMOS cameras I use - whether DSLR or cooled:

https://www.cloudynights.com/topic/636301-asi294mc-calibration-–-testing-notes-thoughts-and-opinions/

This probably applies to any Sony BSI sensor. Obviously, the sensors in Canon DSLRs and mirrorless are not manufactured by Sony but by Canon themselves, yet the guidelines are rational and there is probably at least some similarity between Sony and Canon sensors. For example, the dependence of bias on exposure length also appears to be true of the Toshiba sensor used in the ASI 1600MM (also CMOS, but not BSI) as detailed by Jon Rista in the same thread.

Ultimately, if your images look good to you, there is no reason to change what you are doing.

Yes, I learned the 3second thing the hard way. Anything <1 second on flats was just horrid. Of course a month of rain rolled in before I could get more data points ... maybe I will get a clear night in Feb as Jan is a complete writeoff

I will say I have done 10s, 5s and 1.6s flats and I ended up with the same results in processing and stacking for the most part.

Again, I don't dispute using flats.. i swear by them as I assume most astrophotographers do. My question is still around darks, temperature, etc.

Then - I guess my next question is this:

If my temperature while I am taking my lights is close every time I take images, can I simply use the same darks for around a month as long as the temperature change isn't too drastic?

Remember, the main question I had was the effectiveness of darks since I cannot control the temp and it is going to be different then my lights anyways.

Jared Holloway:
Then - I guess my next question is this:

If my temperature while I am taking my lights is close every time I take images, can I simply use the same darks for around a month as long as the temperature change isn't too drastic?

Remember, the main question I had was the effectiveness of darks since I cannot control the temp and it is going to be different then my lights anyways.

I haven't tried it but I would assume yes. My flats are all over the place (I think it's because of my old lens -- there is moving dust in it) and it's absolutely vital to match them up

I may try doing darks right before imaging as was suggested (only issue is it's right at kids bedtime) or early the next morning (but then I'd have to re-cool the rig or leave it out and its not safe to do so)

Really wish I had a better fenced in yard and a private place to leave things setup

Nick Ambrose:
Jared Holloway:
Then - I guess my next question is this:

If my temperature while I am taking my lights is close every time I take images, can I simply use the same darks for around a month as long as the temperature change isn't too drastic?

Remember, the main question I had was the effectiveness of darks since I cannot control the temp and it is going to be different then my lights anyways.

I haven't tried it but I would assume yes. My flats are all over the place (I think it's because of my old lens -- there is moving dust in it) and it's absolutely vital to match them up

I may try doing darks right before imaging as was suggested (only issue is it's right at kids bedtime) or early the next morning (but then I'd have to re-cool the rig or leave it out and its not safe to do so)

Really wish I had a better fenced in yard and a private place to leave things setup

Yeah, I don't have a lot of dust issues (yet), my main issue for flats is my vignetting (full frame camera shooting through a 42mm t adapter because I can't find a reducer that has a 48mm male connection)

I set up an hour early to take my darks (cover my scope and have a black heavy cloth to cover my image train in case the is light leak) and my temp can go down as far as 8-10 degrees Fahrenheit (4-6 degrees celsius) and I guess that was my main concern in regards to darks. I dont mind taking them, but if there was a way out then I wouldn't. With biases... again, does it hurt to take and use biases? Literally takes me a minute to take 50 so its not a waste of time
I may try to go back to 5s flats and dark flats though (I use a light pad and a double handkerhandkerchief to diffuse the light)

Jared Holloway:
Nick Ambrose:
Jared Holloway:
Then - I guess my next question is this:

If my temperature while I am taking my lights is close every time I take images, can I simply use the same darks for around a month as long as the temperature change isn't too drastic?

Remember, the main question I had was the effectiveness of darks since I cannot control the temp and it is going to be different then my lights anyways.

I haven't tried it but I would assume yes. My flats are all over the place (I think it's because of my old lens -- there is moving dust in it) and it's absolutely vital to match them up

I may try doing darks right before imaging as was suggested (only issue is it's right at kids bedtime) or early the next morning (but then I'd have to re-cool the rig or leave it out and its not safe to do so)

Really wish I had a better fenced in yard and a private place to leave things setup

Yeah, I don't have a lot of dust issues (yet), my main issue for flats is my vignetting (full frame camera shooting through a 42mm t adapter because I can't find a reducer that has a 48mm male connection)

I set up an hour early to take my darks (cover my scope and have a black heavy cloth to cover my image train in case the is light leak) and my temp can go down as far as 8-10 degrees Fahrenheit (4-6 degrees celsius) and I guess that was my main concern in regards to darks. I dont mind taking them, but if there was a way out then I wouldn't. With biases... again, does it hurt to take and use biases? Literally takes me a minute to take 50 so its not a waste of time
I may try to go back to 5s flats and dark flats though (I use a light pad and a double handkerhandkerchief to diffuse the light)

Got it. I assume by "can't find" you mean it exists but not in stock ?

It seems like this might do the job but it's back-ordered

https://www.landseaskyco.com/meade-inst-meade-series-6000-3-inch-flattener-08x.html

I definitely say take biases but I *think* if you have dark flats and darks then they may not be used (i.e. they are included in both the dark flats and the darks, and those are going to be used for calibration)

I *think* the only thing biases are good for is
1. You don't have / need dark flats
2. You want to scale your darks (I guess the same as #1, or you took darks of the wrong length)
i.e. (Dark - Bias) * Scale factor of the two exposure times (Only works on certain cameras)

Nick Ambrose:
Got it. I assume by "can't find" you mean it exists but not in stock ?It seems like this might do the job but it's back-ordered

https://www.landseaskyco.com/meade-inst-meade-series-6000-3-inch-flattener-08x.html

I definitely say take biases but I *think* if you have dark flats and darks then they may not be used (i.e. they are included in both the dark flats and the darks, and those are going to be used for calibration)

I *think* the only thing biases are good for is
1. You don't have / need dark flats
2. You want to scale your darks (I guess the same as #1, or you took darks of the wrong length)
i.e. (Dark - Bias) * Scale factor of the two exposure times (Only works on certain cameras

So, the Meade reducers for the 6000 Series are discontinued... there is an Orion one that I think that will work but its not available until the end of March. Honestly though, in general, I am happy with how my images have processed and where I am at currently with finished fov.

But, if I can skip a night of taking darks I'd be even happier haha

Hi Jared,

Please take a look here:

https://clarkvision.com/articles/dark-current-suppression-technology/

Short answer: you probably don't need darks.

Darks correct 1) dark signal 2) amp glow 3) hot pixels.

(3) is taken care of by dithering and (1) is simply not an issue with a modern DSLR unless your subs are 5-10 minutes (closer to 10 than 5 I'd say) and the ambient temperature something like 25 degrees or more. That leaves amp glow which you can easily check by stretching a single dark frame. If you don't see bright zones at significant parts of the image, then you don't need darks.

Mind you, the result if you use well-matched darks will *always* be better than the result without them if we want to be technical about it, particularly with very faint targets. It is just that the difference will be negligible.

You probably do need flats, because they correct: 1) dust motes 2) uneven responses among your pixels which might manifest as red/green/blue bands and 3) vignetting. You can deal with (3) in the raw development stage, and I guess dithering will mitigate (1) and (2) to some degree. But if you are going to load every frame into Rawtherapee and produce a TIF that is corrected for vignetting, you might as well just use flats and let DSS do everything for you automatically. If you do not have dust motes yes, you can reuse flats as many times as you want.

And if you use flats, make sure you have in-camera dark frame subtraction turned on when you are taking them. This way you will avoid the need for dark flats (technically you can/should use ICDFS for your lights too, but doubling exposure time of a 180 second light is far more disturbing than doubling the exposure time of a 4 second flat).

Regards,
Dimitris

Hi Dimitris,

I used to follow Roger Clark's advice on not taking darks and flats when I first started out and even used his software (RNC color stretch) to process some of my early images before I switched to PixInsight. It worked up to a point. There were issues seen even with recent cameras like my 5D Mark IV (like banding) that I needed calibration frames to over come; I was even in communication with him in regards to one issue, and his advice was to get rid of it in Photoshop. Compare Roger Clark's images (not his astrolandscapes which are excellent, but his DSO images) to some of the good images of the same targets on AB and you'll see that following his advice only takes you so far - the images that reasonably experienced imagers here take are far superior in detail, depth, and color. In camera dark frame subtraction will work, but, as you point out, doubles your exposure time. Roger Clark's advice is not something I'd use past a beginner stage.

Arun

I think, for now, I'll continue to take all of the calibration frames as normal and maybe experiment here and there using all and then not using some to see howcit goes. Thanks for everyone who has responded to this, I really appreciate it!

Hi Arun,

I think we are conflating two different matters:

One is Clark's color managed workflow. This is applicable to very specific gear, namely stock DSLRs with photographic lenses for which there exists at least a DCP profile and preferably a LCP too. It is true that lots of people here are able to produce better detail than Clark, but very few of them are using a stock DSLR and a photographic lens wide open. There are specific tradeoffs in detail vs light gathering with a wide open lens, but really, if you are using that gear I don't see why you shouldn't opt for the advanced raw development capabilities you have at your disposal. For me at least (Nikon D7500) the difference when applying the DCP profile is tremendous.

The other is whether darks are necessary. The link I gave says nothing about the color managed workflow, it simply analyses how dark signal works in sensors with dark current suppression and reaches the conclusion -completely backed by the data- that usually no, they are not.

The two matters are completely unrelated, the only common ground being the hot pixels (which does not really apply if you are dithering). Lots of people with modern DSLRs do not use darks, despite being very "traditional" about their workflow in all other regards. And even some CCD manufacturers advertise "no need to take darks" for their models. The reason being dark signal and amp glows are too low to bother.

Kind regards,
Dimitris

For both my cameras (Canon 6D and 550D) and lenses (Tamron 24-70mm f/2.8, Samyang 135mm) I haven't found yet a software (either the latest version of Lightroom or its open source counterparts) which correctly corrects for vignetting. Without vignetting correction, modelling the light pollution gradients can be very hard. While I understand that DSLRs aren't like the astronomical CCD cameras, I wouldn't go deep enough without doing proper calibration with specialized astronomical image processing software. With all due respect, following Dr. Clark's school of thought, (and of some other well-established astrophotographers who suggest using commercial raw development tools) a beginner will quickly have a decent image to start enjoying his new hobby using the tools he already has, but besides the bright targets, he wouldn't go any further.

I guide usingan ST-4 connection, I use PHD2 for guiding (multi-star) and I use APT for capturing
I dither 20 pixels - 4 in PHD2 and 5 in APT

apt dither of 3 and phd2 of 1 will give you a dither of 16 pixels maximum dither distance, Apt of 2 will give a 11 pixel dither

My calculator here https://drive.google.com/file/d/1ZbMPDTgY-Mp1Oa8SuR1_dytojeg7vsee/view?usp=sharing
download and rename ,values can be edited by using the data validation tool in any spreadsheet program

Related discussions