Trouble with Deep Sky Stacker applying flats correctly - Forums

Topic starter

Dec 9, 2024 · Edited: Dec 9, 2024

I'm having trouble with stacking photos lately and not sure what changed. I'm seeing dust mote artifacts in stacked & calibrated images that shouldn't be there.

Below is an image straight out of DSS. No further editing, STF applied. You can see dust motes in the top left and an overall red/green gradient from the filter. It's more prominent after DBE.

Extra details:

Camera: ZWO ASI2600MC
Lights are 300s
Total lights: 408 frames
Flats taken with a light panel
Flats ADU average around 32,000
Flat batches mostly in the range of 400-900ms (one outlier batch is 3.8s)
Use 40 flats, 30 darks, 30 bias
Everything is grouped by date in the stacking software

What I've tried based on other forum suggestions (none of them worked):

Using dark flats in place of bias.
Resetting DSS settings (Load > restore default settings)
Changing the alignment settings to bilinear instead of automatic

There's several nights of data. About 1/3 of them have lights and flats with those dust motes. The other 2/3 don't, as I cleaned off the sensor. Posted the two different flats & lights below:
https://drive.google.com/drive/folders/1LJgDgo7KcYTBPHdGFOGUQuozpX2Ie5lM?usp=drive_link

Something you'll notice is the dust mote in the flats are very noticeable, but in the light frame, they are really difficult to spot. I'm thinking that may be why? Perhaps that flat is over-calibrating?

There's also a slight rotation between all the frames (no more than 1°), but I doubt that's the issue.

Last comment, I usually avoid stacking in PI because my computer doesn't have the memory, but tried in FBPP. It gave me an error on the debayering process with one image, but I looked at the raw and calibrated version of that image and couldn't find any issues there either. But I did do a stack early on with about 80 frames in FBPP and the image came out just fine. So it's almost like some of the later images are causing issues, but I have no idea how to tell or how to fix.

Helpful Engaging

Julian Shroff

Dec 9, 2024

Could you supply a Light frame and the unstacked flat and dark flat frames? (10 each is enough)

Are you certain that there is no option for a light leak? The red channel of your supplied flat frame is extremely uneven and doesn't follow the circular vignetting flat frames (like your other two channels) usually show.

Helpful Insightful Concise

Derek Vasselin

Topic starter

Dec 9, 2024

Julian Shroff:
Could you supply a Light frame and the unstacked flat and dark flat frames? (10 each is enough)

Are you certain that there is no option for a light leak? The red channel of your supplied flat frame is extremely uneven and doesn't follow the circular vignetting flat frames (like your other two channels) usually show.

Uploaded sample lights, flats, and darks to the Google Drive link above.

I'm pretty confident there's no light leak.

I do have a little bit of tilt in the sensor that I haven't got around to correcting yet, but this was the case long before I started noticing flat issues.

But something I just thought of is I sometimes use the flat panel outside, in the morning. For some of those flats in the folder, you'll see the red on the left side is slightly more aggressive, which I think is the early morning sun (not directly hitting the panel, but bright enough to add some unwanted light on one side).

Helpful Insightful Concise

Julian Shroff

Dec 9, 2024 · Edited: Dec 9, 2024

Derek Vasselin:
Julian Shroff:
Could you supply a Light frame and the unstacked flat and dark flat frames? (10 each is enough)

Are you certain that there is no option for a light leak? The red channel of your supplied flat frame is extremely uneven and doesn't follow the circular vignetting flat frames (like your other two channels) usually show.

Uploaded sample lights, flats, and darks to the Google Drive link above.

I'm pretty confident there's no light leak.

I do have a little bit of tilt in the sensor that I haven't got around to correcting yet, but this was the case long before I started noticing flat issues.

But something I just thought of is I sometimes use the flat panel outside, in the morning. For some of those flats in the folder, you'll see the red on the left side is slightly more aggressive, which I think is the early morning sun (not directly hitting the panel, but bright enough to add some unwanted light on one side).

Sorry it seems that I misunderstood and thought you took dark flats instead of bias frames, please upload the bias frames as they are relevant for flat frame calibration

Concise

Dec 9, 2024 · Edited: Dec 9, 2024

Derek Vasselin:
I'm having trouble with stacking photos lately and not sure what changed. I'm seeing dust mote artifacts in stacked & calibrated images that shouldn't be there.

Below is an image straight out of DSS. No further editing, STF applied. You can see dust motes in the top left and an overall red/green gradient from the filter. It's more prominent after DBE.

Extra details:
Camera: ZWO ASI2600MC
Lights are 300s
Total lights: 408 frames
Flats taken with a light panel
Flats ADU average around 32,000
Flat batches mostly in the range of 400-900ms (one outlier batch is 3.8s)
Use 40 flats, 30 darks, 30 bias
Everything is grouped by date in the stacking software

What I've tried based on other forum suggestions (none of them worked):
Using dark flats in place of bias.
Resetting DSS settings (Load > restore default settings)
Changing the alignment settings to bilinear instead of automatic

There's several nights of data. About 1/3 of them have lights and flats with those dust motes. The other 2/3 don't, as I cleaned off the sensor. Posted the two different flats & lights below:
https://drive.google.com/drive/folders/1LJgDgo7KcYTBPHdGFOGUQuozpX2Ie5lM?usp=drive_link

Something you'll notice is the dust mote in the flats are very noticeable, but in the light frame, they are really difficult to spot. I'm thinking that may be why? Perhaps that flat is over-calibrating?

There's also a slight rotation between all the frames (no more than 1°), but I doubt that's the issue.

Last comment, I usually avoid stacking in PI because my computer doesn't have the memory, but tried in FBPP. It gave me an error on the debayering process with one image, but I looked at the raw and calibrated version of that image and couldn't find any issues there either. But I did do a stack early on with about 80 frames in FBPP and the image came out just fine. So it's almost like some of the later images are causing issues, but I have no idea how to tell or how to fix.

You have Pixinsight!

I recommend to stop using DSS, you're limiting yourself!

Go into scripts > batch processing > Weighted batch preprocessing, and learn how to use that, it's very straight forward, and can perform a wide variety of methods for stacking.

Go into post-calibration, and turn on drizzle integration, 1 scale, varshape 1, 0.85 dropshrink, and grid size 16. It'll take a little bit longer than DSS but the result will be extremely sharp.

Helpful Engaging Supportive

Alex Nicholas

Dec 10, 2024 · Edited: Dec 10, 2024

if you cleaned your sensor mid imaging process, you need to calibrate the 'uncleaned sensor images' with 'uncleaned sensor flats' and the 'cleaned sensor' images with 'cleaned sensor flats'. Then integrate the calibrated frames..

You won't be able to do this in WBPP I don't believe, but the upside is that you only need to calibrate the frames separately, outside of WBPP, then point WBPP to your calibrated frame directory, and let it do the rest, ignoring any calibration steps…

if you don't have flats for the rig after cleaning the sensor, you're going to have issues. Even if its only 1/3 of the total dataset that has no dust motes, what if that was the night with the best seeing, and the stacking algorithm is therefore, weighting those frames more heavily than the properly calibrated frames. The frames where the light does not have the dust mote, but the flat you're using is correcting it, are going to contribute more to the final image, despite having overcorrected areas, which is going to result in an image like what you have shown, where you have 'inverse dust motes'

In future, especially if you're planning on shooting LONG integration projects like this.

Clean your sensor/filters before the first night of imaging, and then do your flats. image for multiple nights, do not disassemble your camera/scope if you do not 100% have to. finish your last night of imaging, and take flats again (the same amount, and the same duration as before night one).

By doing so, your flat will be an average of 'how the flat was at the start of the project' and 'how the flat is at the end of the project'. This is going to yield 2nd most accurate flat calibration you can have…. The most accurate would be shooting flats at the start and end of each night.. Calibrate the images taken each night with the flats from that particular night, then throw away the flats, and do the same for all consecutive nights of the project…

I would never pull apart my rig and clean the sensor/filters half way through a large imaging project, as there is such a high chance for calibration failures, rotation issues, introducing differing tilt between sessions etc..

Get it clean before undertaking a big project, and do everything you can to not have to open the optical train at all.

Helpful Insightful

Derek Vasselin

Topic starter

Dec 10, 2024

Alex Nicholas:
if you cleaned your sensor mid imaging process, you need to calibrate the 'uncleaned sensor images' with 'uncleaned sensor flats' and the 'cleaned sensor' images with 'cleaned sensor flats'. Then integrate the calibrated frames..

You won't be able to do this in WBPP I don't believe, but the upside is that you only need to calibrate the frames separately, outside of WBPP, then point WBPP to your calibrated frame directory, and let it do the rest, ignoring any calibration steps...

if you don't have flats for the rig after cleaning the sensor, you're going to have issues. Even if its only 1/3 of the total dataset that has no dust motes, what if that was the night with the best seeing, and the stacking algorithm is therefore, weighting those frames more heavily than the properly calibrated frames. The frames where the light does not have the dust mote, but the flat you're using is correcting it, are going to contribute more to the final image, despite having overcorrected areas, which is going to result in an image like what you have shown, where you have 'inverse dust motes'

In future, especially if you're planning on shooting LONG integration projects like this.

Clean your sensor/filters before the first night of imaging, and then do your flats. image for multiple nights, do not disassemble your camera/scope if you do not 100% have to. finish your last night of imaging, and take flats again (the same amount, and the same duration as before night one).

By doing so, your flat will be an average of 'how the flat was at the start of the project' and 'how the flat is at the end of the project'. This is going to yield 2nd most accurate flat calibration you can have.... The most accurate would be shooting flats at the start and end of each night.. Calibrate the images taken each night with the flats from that particular night, then throw away the flats, and do the same for all consecutive nights of the project...

I would never pull apart my rig and clean the sensor/filters half way through a large imaging project, as there is such a high chance for calibration failures, rotation issues, introducing differing tilt between sessions etc..

Get it clean before undertaking a big project, and do everything you can to not have to open the optical train at all.

I don't follow? DSS and PI can both calibrate lights / flats by group.

I took flats each night (before and after the sensor cleaning), so the calibrations are grouped by night.

Alex Nicholas

Dec 10, 2024

Is it actually doing that though? because your result looks to me like light frames that did not exhibit dust motes were calibrated with flats that did have dust motes..
I'm not seeing over correction in the image corners, or a significantly dark central region… just bright spots that look like over corrected dust motes.

Derek Vasselin

Topic starter

Dec 11, 2024

Julian Shroff:
Derek Vasselin:
Julian Shroff:
Could you supply a Light frame and the unstacked flat and dark flat frames? (10 each is enough)

Are you certain that there is no option for a light leak? The red channel of your supplied flat frame is extremely uneven and doesn't follow the circular vignetting flat frames (like your other two channels) usually show.

Uploaded sample lights, flats, and darks to the Google Drive link above.

I'm pretty confident there's no light leak.

I do have a little bit of tilt in the sensor that I haven't got around to correcting yet, but this was the case long before I started noticing flat issues.

But something I just thought of is I sometimes use the flat panel outside, in the morning. For some of those flats in the folder, you'll see the red on the left side is slightly more aggressive, which I think is the early morning sun (not directly hitting the panel, but bright enough to add some unwanted light on one side).

Sorry it seems that I misunderstood and thought you took dark flats instead of bias frames, please upload the bias frames as they are relevant for flat frame calibration

I uploaded dark flats and biases to that original google drive folder.

I have done both (not at the same time). Originally, I used biases. Then thought that might have been the issue, so I stacked a second time with dark flats in place of biases. Same result either way.

Derek Vasselin

Topic starter

Dec 11, 2024

Alex Nicholas:
Is it actually doing that though? because your result looks to me like light frames that did not exhibit dust motes were calibrated with flats that did have dust motes..
I'm not seeing over correction in the image corners, or a significantly dark central region... just bright spots that look like over corrected dust motes.

That's what I'm trying to figure out. It's supposed to calibrate the lights from night 1 with flats from night 1, then lights from night 2 with flats from night 2, and so on.

But I agree, it does look like that's whats happening, and I've triple-checked that I'm grouping everything correctly.

Rick Veregin

Dec 11, 2024

Derek Vasselin:
I'm having trouble with stacking photos lately and not sure what changed. I'm seeing dust mote artifacts in stacked & calibrated images that shouldn't be there.

Below is an image straight out of DSS. No further editing, STF applied. You can see dust motes in the top left and an overall red/green gradient from the filter. It's more prominent after DBE.

Extra details:
Camera: ZWO ASI2600MC
Lights are 300s
Total lights: 408 frames
Flats taken with a light panel
Flats ADU average around 32,000
Flat batches mostly in the range of 400-900ms (one outlier batch is 3.8s)
Use 40 flats, 30 darks, 30 bias
Everything is grouped by date in the stacking software

What I've tried based on other forum suggestions (none of them worked):
Using dark flats in place of bias.
Resetting DSS settings (Load > restore default settings)
Changing the alignment settings to bilinear instead of automatic

There's several nights of data. About 1/3 of them have lights and flats with those dust motes. The other 2/3 don't, as I cleaned off the sensor. Posted the two different flats & lights below:
https://drive.google.com/drive/folders/1LJgDgo7KcYTBPHdGFOGUQuozpX2Ie5lM?usp=drive_link

Something you'll notice is the dust mote in the flats are very noticeable, but in the light frame, they are really difficult to spot. I'm thinking that may be why? Perhaps that flat is over-calibrating?

There's also a slight rotation between all the frames (no more than 1°), but I doubt that's the issue.

Last comment, I usually avoid stacking in PI because my computer doesn't have the memory, but tried in FBPP. It gave me an error on the debayering process with one image, but I looked at the raw and calibrated version of that image and couldn't find any issues there either. But I did do a stack early on with about 80 frames in FBPP and the image came out just fine. So it's almost like some of the later images are causing issues, but I have no idea how to tell or how to fix.

The issue seems to be in the raw flats. Here is what the fits flat looks like when I opened it in my fits viewer, with an autostretch.

Your flat should be flat in color. With an OSC camera you should see a uniform green, though the green hue can change in a vignette type pattern due to the difference in brightness with a strong stretch. This is where your color gradient is coming from the raw flat itself. This has nothing to do with any settings, bias, darks, just the flat itself.

There is either a light leak, remember this is a strong stretch it could be a small leak. Or your light panel is not uniform in color intensity.

I have a mask for my light panel, in my case it covers everything but the part of the panel over my dew shield. This also acts to sort of seal the edge of the panel against the dew shield to any light leak. I then do my flats in reasonably dark conditions (my garage), before I wheel my telescope out. So little opportunity for any stray light.

Did you try doing 10 flats, then rotated the panel 90 degrees, do 10 more, and then do the same at 180 and 270 degrees, then average them? Might not work perfectly, but you can see if it improved your flat color gradient.

You could also try the famous t-shirt flat, I have used that in my garage just with the ambient light, not pointing near the light source, and it works fine too. Or against the sky, but that can be tricky near sunset in particular, you should point straight overhead where color gradients are least.

Once you can get a raw flat that is just green everywhere, your problems should go away. I think the overcorrection (the donuts are bright, so it is overcorrecting) is because the green has faded in that corner, so you are getting an extra green boost in the corner--again green dominates the flat.

I can't think of any other reasons, but I am reasonably sure it is the raw flat you must fix.

Hope this helps
Rick

Helpful

Derek Vasselin

Topic starter

Dec 13, 2024 · Edited: Dec 13, 2024

Rick Veregin:
Derek Vasselin:
I'm having trouble with stacking photos lately and not sure what changed. I'm seeing dust mote artifacts in stacked & calibrated images that shouldn't be there.

Below is an image straight out of DSS. No further editing, STF applied. You can see dust motes in the top left and an overall red/green gradient from the filter. It's more prominent after DBE.

Extra details:
Camera: ZWO ASI2600MC
Lights are 300s
Total lights: 408 frames
Flats taken with a light panel
Flats ADU average around 32,000
Flat batches mostly in the range of 400-900ms (one outlier batch is 3.8s)
Use 40 flats, 30 darks, 30 bias
Everything is grouped by date in the stacking software

What I've tried based on other forum suggestions (none of them worked):
Using dark flats in place of bias.
Resetting DSS settings (Load > restore default settings)
Changing the alignment settings to bilinear instead of automatic

There's several nights of data. About 1/3 of them have lights and flats with those dust motes. The other 2/3 don't, as I cleaned off the sensor. Posted the two different flats & lights below:
https://drive.google.com/drive/folders/1LJgDgo7KcYTBPHdGFOGUQuozpX2Ie5lM?usp=drive_link

Something you'll notice is the dust mote in the flats are very noticeable, but in the light frame, they are really difficult to spot. I'm thinking that may be why? Perhaps that flat is over-calibrating?

There's also a slight rotation between all the frames (no more than 1°), but I doubt that's the issue.

Last comment, I usually avoid stacking in PI because my computer doesn't have the memory, but tried in FBPP. It gave me an error on the debayering process with one image, but I looked at the raw and calibrated version of that image and couldn't find any issues there either. But I did do a stack early on with about 80 frames in FBPP and the image came out just fine. So it's almost like some of the later images are causing issues, but I have no idea how to tell or how to fix.

The issue seems to be in the raw flats. Here is what the fits flat looks like when I opened it in my fits viewer, with an autostretch.

Your flat should be flat in color. With an OSC camera you should see a uniform green, though the green hue can change in a vignette type pattern due to the difference in brightness with a strong stretch. This is where your color gradient is coming from the raw flat itself. This has nothing to do with any settings, bias, darks, just the flat itself.

There is either a light leak, remember this is a strong stretch it could be a small leak. Or your light panel is not uniform in color intensity.

I have a mask for my light panel, in my case it covers everything but the part of the panel over my dew shield. This also acts to sort of seal the edge of the panel against the dew shield to any light leak. I then do my flats in reasonably dark conditions (my garage), before I wheel my telescope out. So little opportunity for any stray light.

Did you try doing 10 flats, then rotated the panel 90 degrees, do 10 more, and then do the same at 180 and 270 degrees, then average them? Might not work perfectly, but you can see if it improved your flat color gradient.

You could also try the famous t-shirt flat, I have used that in my garage just with the ambient light, not pointing near the light source, and it works fine too. Or against the sky, but that can be tricky near sunset in particular, you should point straight overhead where color gradients are least.

Once you can get a raw flat that is just green everywhere, your problems should go away. I think the overcorrection (the donuts are bright, so it is overcorrecting) is because the green has faded in that corner, so you are getting an extra green boost in the corner--again green dominates the flat.

I can't think of any other reasons, but I am reasonably sure it is the raw flat you must fix.

Hope this helps
Rick

Hey Rick

I'm confident this isn't a light leak. I've done several tests on that, including what you suggested. The same result with flats, no matter what.

I believe this gradient results from some sensor tilt, which also hits the filter at a slight angle. I just haven't gotten around to fixing it yet.

But the flats have always looked this way (well before I started noticing issues). And the lights show the same gradient.

It's definitely overcorrecting. I just can't figure out why.

Rick Veregin

Dec 13, 2024

Derek Vasselin:
Hey Rick

I'm confident this isn't a light leak. I've done several tests on that, including what you suggested. The same result with flats, no matter what.

I believe this gradient results from some sensor tilt, which also hits the filter at a slight angle. I just haven't gotten around to fixing it yet.

But the flats have always looked this way (well before I started noticing issues). And the lights show the same gradient.

It's definitely overcorrecting. I just can't figure out why.

Well the purpose of flats is to do just that, correct for problems such as this. The problem is if it is too severe a problem with your optical path, or you need to stretch your image more, the flats need to perfectly correct your lights, which can be difficult given the tolerance will be low. That one side has very little green, so the correction is very large. This can result in either over or undercorrection. I believe the undercorrection is due to the lack of green signal where those donuts are.

For example, after a few minutes exposure may look like the image is flat, but when you integrate over hours the image flatness degrades. This is because you can stretch the image more with more integration as the noise is low, and this shows every flaw in your calibration. I have found cases where the dust donut is just too opaque, the flats that worked fine before, now show a bit of donut (over or under) again especially at very long intergration with low noise. The solution is to clean the sensor at that point.

If it is a combination of the filter and a tilt, then the light source (target&sky vs flat panel) characteristics can be very important. As an extreme example, if you flat light source were very green and your sky and image were red say, then the flat would not see the same light characteristics and thus the flat would not properly calibrate the light, especially since you have two green pixels for every red and every blue pixel, and green typically has the highest transmission as well in the Bayer filters.

When you say you believe it is tilt, if you were to turn your camera say 180 degrees and image your target, the color shift in your light frames should also flip 180. You might try that to confirm this is the problem, if you haven't.

Anyway, if tilt is the problem, fixing that is the solution, and then your flats will look normal and should work. Your flat itself is definitely not normal, you should not expect these flats to work that well the way it is with such a severe color gradient.
Rick

Helpful Insightful Respectful

Haakon Rasmussen

Dec 13, 2024

I have had similar problems. It looks like DSS is not capable of stacking several nights together with different Flats and Dark Flats. I have solved the problem by only stacking one night at the time, and making DSS save each light frame as a calibrated frame. Then i stack all the calibrated frames together. You will find this setting under intermediate files. Try it out, perhaps it will work for you.

CS Haakon

Helpful Concise Supportive