Back to forums

Help with Flats Overcorrecting, Please

Sky-Watcher Star Adventurer Mini Pleiades Astrophoto PixInsight Processing Calibration Deep Sky
19 replies•353 views
Clint avatar
Clint avatar
Hi,
So for the past few months I've been experiencing an issue where my flats are now overcorrecting my lights. When the issue started, I had not made any changes to how I acquired or processed the flats (or the rest of the data). Since then, however, I have tried everything that I could think of or find to try and resolve the problem. There is just some blindspot that I have here and I could really use y'alls' help.


TL;DR here is some recently acquired data for you to take a look at: please help me with over-calibration
In the shared dropbox are all of my RAWs as well as the Pix Insight generated master calibration files.
(screenshot from pixinsight of a corrected light sub and the master flat -- STF applied to both)


Want more info?

So I shoot with just a Nikon D7100 and Tamron 70mm-200mm lens on a Star Adventurer Mini Pro.
I primarily use PixInsight for processing.

For my flats, I strive to get to half well depth as measured in PixInsight via Image Statistics. This was working well until February. Then the over calibration issue magically appeared.

I was calibrating the flat with both Bias and matched dark flats. Again, this was working. I was not seeing any over calibration in the lights.
Now, I can't even mitigate the over calibration in pre processing.

I'd really like to keep using flats calibration as there is a lot of dust in my optical train.

Things I have tried:
Uncalibrated Flats
Flats that are Bias-calibrated only
Flats that are Bias & Dark Flat calibrated (as mentioned above)
Flats that are Dark Flat calibrated only
Flats that are Bias & optimized dark calibrated

Lights that are bias and flat corrected
Lights that are flat corrected only
Lights that are fully corrected (flat, bias, dark)

Any and all help is much appreciated!
here's the dropbox link again if you've made it this far

Clear Skies,
Clint
Helpful Engaging
andrea tasselli avatar
andrea tasselli avatar
Just for starters: how do you acquire the flats? That's the key issue, afaik. Is the optical train exactly the same given that you're using a zoom lens? Changed the f-stop maybe? Same focal length? In the interim you could try to multiply the flats (bias subtracted only) by a factor (start with 0.5) and see how well the light frame is corrected. I'll have a look at your data but it may take a while…
Helpful Insightful Respectful Engaging Supportive
Clint avatar
Clint avatar
Hi @andrea tasselli,
flats are acquired in a 'dark room' (e.g. closet) with a tracing panel and diffuser on the lens objective. to the best of my ability the optical train does not change. I always confirm camera/lens settings before taking the flats and pixinsight is pretty good at telling you if you goofed on the basics like focal length, aperture and iso. This process had worked for me for a year and suddenly things went wonky.

So, I think what your saying is take a bias-only corrected master flat and using PixelMath multiply it by 0.5 and use the adjusted flat for lights calibration to see what happens? Just confirming that I understood correctly....

Appreciate you having a look at the data!

Clear Skies,
C
andrea tasselli avatar
andrea tasselli avatar
Hi Clint,

Yes, that is exactly that. I never take flats the way you do it but that doesn't mean yours is wrong.
Clint avatar
Clint avatar
lol. can't take sky flats. covering a zoom lens with a t shirt will absolutely mess up the optical train (focus usually) I tried so many times to make that work. don't have an observatory so can't have a static wall based solution. tracing panel is too big to just hold in front of the camera in situ. this was the best compromise I could devise.

thanks. will try adjusting the master flat and see what happens.
Die Launische Diva avatar
Die Launische Diva avatar
Hello @Clint, what happens if you reuse an older master flat for which you know it worked well in a different imaging session?
Clint avatar
Clint avatar
@Die Launische Diva interesting idea. will have to dig out some old data though. this data was acquired with a new lens at f/3.2 and my old lens only went as fast as f/4.5 at this focal length. The good news is the issue exists across lenses so I can definitely try that out.
Clint avatar
Clint avatar
@andrea tasselli assuming I find the 'right' amount to reduce the flat by, does that indicate I could take my median ADU (as measured by PixInsight) and multiply it by the same factor to determine the 'proper' exposure time needed?
andrea tasselli avatar
andrea tasselli avatar
Yes. The purpose is that. I went through the calibration procedure and I can't find any obvious fault. The issue is still the way you take flats. I'd venture to suggest to try sky flats or use a fluorescent panel just right after your imaging session is over, which is always good practice with DSLR (dust might move around when you flip the mirror). BTW, I can see very little signs in your light frames, except for stars obviously.
Clint avatar
Clint avatar
I'll see what I can do for in situ flats.
Yeah @ 40seconds ,NGC7000 is going to be quite dim and there will be a lot of work pulling it out. It is there though (lower right quadrant)

Thanks for taking a look and for your advice!
andrea tasselli avatar
andrea tasselli avatar
I am through the 8th iteration and still not got to the bottom of it, which is puzzling. I'll have to look at the light frames before going any further. Apparently you have a very flat centre field in your flats (pun unintended), which is odd as it should broadly follow a gaussian profile. 2 seconds seems to be way too much for a flat with artificial light illuminating it, the way I see it. I am at 1/25s and never had to complain. With faster optics I'd probably stick to 1/100s at 200 ISO (we both use Nikons).
Helpful
Clint avatar
Clint avatar
Weirder and weirder. I’ve actually tested with another D7100 camera body to make sure I didn’t have some unknown problem with the sensor and the problem is there too. I tested it with a different lens on the same body and it was there too. I’m really, really stumped.  

When the problem first popped up I bought an on-lens diffuser that only let’s in 18% of the light to make sure there wasn’t some weirdness going on with the light panel or shutter shadow and that’s why they’re so long right now.

If you look at my gallery, anything pre February didn’t have this problem as far as I can tell.
andrea tasselli avatar
andrea tasselli avatar
Even at 5% the illumination is wrong so it isn't just a factor, is a different illumination entirely.  I synthesised what the illumination profile for your inlet pupil would look like (sans dust spots, obviously) and you can compare it to your flat. Your flat is much much flatter than that. Having ruled out methodology, lens and camera the only remaining viable explanation is flat acquisition methodology. If it is clear and you can do it take a sequence of 5s shots with an interval of 1 min between each shot when you stop the sidereal drive. Point the lens toward the zenith to be sure you get not a lot of moon illumination. If it  is cloudy even better, take a series of 3 min shots (say 20-30) not tracked. Use a median integration (after the usual preprocessing but not flat fielding) and a strong rejection filter in PI and you should obtain a fairly decent and hopefully permanent flat field. Do not debayer these frames.

See below for your pupil illumination.

Helpful
andrea tasselli avatar
andrea tasselli avatar
To make the point even clearer here are the two flats (original and recovered from the light frames) side by side using the same STF:
Clint avatar
Clint avatar
Wow Andrea, just wow. Thank you for spending so much time and effort looking into my problem.
So, it’s not _just_ that the flats are over exposed (certainly looks that way from your comparison) but also something is happening when i move to take flats that is really disturbing the optical train.

Plenty of cloudy nights here so taking the over long exposures as you recommend won’t be an issue. Doing it on the same night where the skies are clear at some point too will be a little tricky.  Not going to try tonight as the moon is still very bright, but I will give it a go at some point.

A friend led me to this post on Cloudy Nights. Have you seen it?

Clear Skies!
Respectful Supportive
andrea tasselli avatar
andrea tasselli avatar
Yes, quite interesting but rather convoluted as a method. What you want is a sure-fire way to get a decent flat every time you need it. With lenses I take the flat after the session is over (and it is rather taxing affair doing them in the wee hours) using a FL panel. Sky flat work even better but needs to be done at dusk or using LP illumination of the cloudy sky (if you have clouds bothering you during a session).

There is anther method that I had not implemented in PixelMath as I can't be bothered to work out what the right commands are (what PAID software is there with no help files??). Arrive at the synthetic flat (i.e. proceed with the usual pre-processing but without flat-fielding, de-bayering and alignment) and stack these files up. Use DBE to retrieve a synthetic flat. Use the synthetic flat to scale the "real" flat to the correct inlet pupil illumination by normalising against the pixel per pixel value of the synthetic flat normalised (adjustment may be necessary to work out what is the best strategy here, My assumptions that is best to scale against  the normalised max minus local-pixel-value).
Clint avatar
Clint avatar
Definitely not a method I would want to use all the time. But might be useful in figuring out the issue I am having right now.

And OMG. Nearly $300 USD for this software and no decent help files. I think I remember seeing them say in their forums that is was an impossible task??? I’ve worked in SaaS for 20+ years and customer edu for the last 10.  You always write documentation! Lowest cost way to correctly support your users.

Thanks again for all of your help!
Jérémie avatar
Jérémie avatar
andrea tasselli avatar
andrea tasselli avatar
Clint:
Definitely not a method I would want to use all the time. But might be useful in figuring out the issue I am having right now.

And OMG. Nearly $300 USD for this software and no decent help files. I think I remember seeing them say in their forums that is was an impossible task??? I’ve worked in SaaS for 20+ years and customer edu for the last 10.  You always write documentation! Lowest cost way to correctly support your users.

Thanks again for all of your help!

Not only that (there is some help but not really extensive or up to date), most if not all the support is essentially left to the users' community. I basically gave up on that and most of the times I try to work out a solution for myself. Which I think is what they had in mind all along...
Daniel Erickson avatar
Daniel Erickson avatar
Jérémie:
Check this video from @Adam Block 

https://www.youtube.com/watch?v=Kkg441UBNpo

https://pixinsight.com/forum/index.php?threads/flat-field-over-correction.15595/


Thanks, @Jérémie for these links. I don't use PI, but the methodology of problem-solving and explanatory power of the lesson is quite good.