WBPP Script seems to have gone off the rails

Hi Michael,
For what it's worth whenever I see something like that, my immediate thoughts are that I forgot to either to set the right gain and offset for either my flats or lights in my acquisition sequencer, or I chose the wrong darks or flats in putting WBPP inputs together.   I have done both.   All can be fixed.   

Recently, using NINA, I forgot to set my Offset (I always use "50" for my camera = 500ADU for a Zwo 16bit) and it defaulted to an offset to 1 (or 10ADU).    This made the calibrated lights look very dark after WBPP, which used a flat taken with an offset of 50 on a light frame with an offset=1 - resulting in 490 ADU subtracted from the calibrated lights, making them very dark and clipped.  Fortunately the situation was repairable by using Pixelmath and Image container to add the offset ADUs back into the light frame before calibration with WBPP.   In your case, I would make sure you selected the correct and identical gain/offset files in both your manual and WBPP calibrations.  

Hope this help - sorry if I am completely offbase.

Dave

Whenever something like that happens to me I will rerun without flats just to do a process of elimination.

Yes. Local normalization was set.

Daniel. Thanks for the suggestion. I will try this out and let you know.

Michael,

I followed the link to your issue posting from the collaboration of your data with Uwe.  I wasn't a member of this forum so hadn't seen it before.  Reading your OP was deja vu all over again.  I apologize in advance for the length of this reply, but this covers an issue I spent a couple of weeks in getting to a solution.

I had a similar/same issue last fall where my stacked images were almost flat in data after stacking despite looking very good unstacked.  As I dug into this, I found the stacked histogram reduced significantly vs the unstacked images, especially on the Ha master.  I went through every WBPP setting I could think of trying to understand what was happening.  I turned the problem with some data files over the PI support forum and got some feedback very quickly.  My darks had a higher ADU level than my black level on my data files.  In other words, light leak on my darks.

It was one of those "…duhhh…" moments; I had been in the habit of inspecting the high/low rejection frames in WBPP.  Particularly the Low rejection - that frame is the "canary in the coal mine" indicator of bad calibration frames, particularly Darks.  If you see much of anything there, you have some calibration file problems.  Unfortunately, with the addition of the drizzle function ( which I'm addicted to), the Hi/Lo rejection frames are not part of the display for the drizzled master frame when first opened.  So you have to open the non drizzled masters to access the Hi/Lo rejection frames to check for issues.

As it turns out, I have a light leak, not surprising really - 300 sec exposures (or longer) for darks are going to find anything short of perfection if you are shooting Darks during daylight hours with just OTA covers.

It isn't really practical to remove the camera and install the OEM sensor cover (for me) since that affects the tilt adjustments which are important on APS-C size sensors and larger.  So while I also went and taped every seam and gap/crevice anywhere in the optical train, I also changed to shooting Darks after full darkness and with my Telegizmo 360 cover in place to get as close to absolute darkness around the OTA/imaging trains as I can.

That did the trick and provided the optimal solution to what I thought initially was some weird problem with the imaging files or WBPP.  However, I was able to "correct" a mistake I made in WBPP to allow me to progress until I had the calibration files 'fixed'.   That is covered below as it touches on another issue specific to CMOS sensors and NB filter imaging.

A quick check for light leaks is to check your darks (or master dark) and get the mean ADU and compare that to the mean ADU of unprocessed image files.  If the level of the dark is almost equal or higher than that of your lights,  you are going to get data clipping.  However, there is another issue that can give similar results and has a solution embedded in your camera and WBPP settings.

This is the oft reported problem of insufficient offset or lack of pedestal in your lights when using a CMOS camera and NB filters

You can find an in depth discussion with Adam Block on his YouTube channel and there are some Pixel Math scripts to analyze your light frame to determine if this is part of the problem.  (if there are absolute black/negative value pixels in your image, you have this problem.  Increasing the offset in the cameras settings, or adding pedestal in WBPP can compensate for this.  WBPP will actually analyze the light frames and add an appropriate amount of pedestal, IF that option is selected.  However, the default is to NOT perform the automatic pedestal function so you have to remember to select it each time.  

The thing to remember is having WBPP perform an automatic pedestal correction can also protect your processing from MINOR light leak induced clipping, as well as protecting you from the infamous CMOS/NB clipping problem.  If I remember correctly, I also saw some light leak influence in my FlatDarks, but not to the extent that showed up in my Darks.  Pedestal won't really address much in there is a problem with the FlatDarks, but the effects of bad calibration induced clipping will always show up in the Hi/Lo rejection images, at least that has been my experience.

Looking at the stats you provided in your posting, you didn't show the stats for your calibration frames/darks.  However, from comparing your unprocessed results to the stacked results, it seem clear that your Darks have an ADU level close to equal to that of your Ha light frames since the stacked Ha master has an ADU mean of close to zero instead of a healthy level of pedestal shown on the Sii, and Oiii.  That (to me) is an indicator there is an issue with your calibration files.

Uwe's post with his processed image of your data indicated he didn't do anything different in his WBPP setup, yet he didn't have your issues.  I would be willing to bet his standard WBPP set either has a very healthy fixed pedestal value or he normally selects the automatic pedestal adjustment.  Pedestal adjustment can correct for a light leak/higher ADU value in the DARKS to an extent (that is what I did first until I could shoot a new set of darks).

I believe if you just perform a 'quick' WBPP run with auto pedestal enabled, you'll find that most of your issue/loss of data in stacking is resolved.  And/ or you can check the adu level of the Ha darks/dark master vs the Ha lights to also check if they are close/equal/greater than that of the Lights.

Michael J. Mangieri:
Thanks George, and as I commented in the conversation attached to the image I'll do that run in PI with auto pedestal enabled and also check my darks.

Update:  My master darks (for all exposure values) has a mean ADU of 500.

That means you would have used an offset/pedestal of 50 when taking your darks, the default value on ASI cameras.