TLDR: Fun experiment, example image and links to scripts at bottom of post!
A couple of weeks ago Dirk Bausch put up a post about creating a pipeline that followed the ABYSS process (Borlaff et al. 2019, A&A 621, A133) for background removal on a per-sub basis. It was a very interesting experiment and some great discussion was had. Since I live in a Bortle 7/8 and am generally super lazy with scheduling my imaging to avoid things like a bright moon, I thought this method might actually work well for me.
This method of removing backgrounds on a per-sub basis should in principle be good for variable gradients throughout the course of the night (state of the moon, general change in gradients as altitude of target changes, etc…) and across multiple nights (changes in high altitude thin cloud cover, seeing, transparency, etc…)
Because the last time I coded anything substantial was in college and I have forgotten most of the FORTRAN / IDL I was using back then, I had Gemini help me create a Bash / Python script that employs the method. My OS is Kubuntu 26.04 LTS running X11 KDE Plasma Desktop. The script that I used Gemini to develop utilizes various GNU Astronomy Utilities to employ the ABYSS strategy as outlined in the Borlaff paper. Though Gemini did the coding, I have enough background in programming and of using Linux that I could sanity check the work as the script was built. I definitely welcome constructive criticism and advice though!
Outline of pre-pipeline (for OSC)
WBPP in PixInsight to do calibration and cosmetic correction (debayering also happens but those files don’t get used - it is simply so flat frames are properly selected.)
Blink the cosmetically corrected frames to remove ones with obvious issues
Use SplitCFA in PixInsight to separate the sub frames into individual pixel channels at half resolution
Depending on whether the target is an extended nebula or galaxies / planetary nebula / cluster, register the highest SNR CFA frames (extended nebula) or register all CFA frames (galaxies…) and put them in the same folder as the unregistered SplitCFA frames.
some of the later features I added to the script rely on the frame that is used to do the registering be plate solved. Best practice is to plate solve the cosmetically corrected frame intended for registration prior to the CFA split
Batch format convert the .XISF files to .fits (required for the pipeline)
Employing the Pipeline
Navigate in the terminal to the folder that contains the .fits SplitCFA files and their registered counterparts and run the script
worth noting at this step that I actually developed 2 scripts, the main one that does the processing, and also a watchdog script that runs as a wrapper around the first that detects if the first script hangs, then cleans things up, and restarts the processing
When I run the script, I actually run the watchdog, which then in turn runs the primary pipeline
When completed, the script will have performed per-sub background removal and will have reconstructed the SplitCFA frames into Bayered images
Back to PixInsight
Debayer the frames produced by the pipeline
Register the frames (does not need to be the same registration that was done prior to the pipeline) including the generation of drizzle files
Integrate the registered frame with drizzle files
CFA Bayer drizzle (I experimented with 1x and 2x and actually got better results with 1x drizzle, though your mileage may vary)
my specific settings are 1x drizzle, 1.0 drop shrink, CFA checked, circular kernel shape. These settings are simply what works for my data, your settings may depend on your specific situation
Continue with your normal processing workflow
Final Output
At this point you should have a 1x CFA drizzle integration that gives you an un-interpolated debayered color image (monochrome folks will have already adjusted the workflow by this point)
A clean up of files can happen here, so long as you keep the recombined bayered frames, the debayered frames, and the registered frames (the ones after recombining the CFA images) you can go through the drizzle process again with different settings and everything the drizzle integration script needs will be there. Though do note that if you are utilizing the iterative nature of the script, as described below, then you will want to keep the files that are coming out of the pipeline until the project is done.
Other notes on the script:
This has been developed to be an iterative process. Because I have been under constant clouds, I have been testing this on my most recent full data sets (~200 - 300 total good frames) so overall this has been taking a long time, but the script works by checking relevant folders for new frames and skips frames it has already completed. As a normal part of a workflow, this process shouldn’t add too much time if you use it on a single night’s data set.
Personal Conclusions:
In the original thread, there was a lot of great discussion and friendly critique of the methods, comparisons, and efficacy of the process. This is purely anecdotal, but here are some conclusions I have reached after running through this pipeline a few times:
Once you find the right settings, the per-sub BG removal definitely works, and works well
Gradients in the final CFA Drizzle integration are still present, but are much simpler to correct
Low SNR structures are well preserved and are more easily preserved when using BG removal tools later in the workflow
Artifacts and large scale noise structures are greatly reduced (noise reduction works a lot better with less aggressive settings)
SPCC graphs are some of the tightest I have seen (at least for my data)
This doesn’t work miracles but it does improve the quality of the integrated stack
Even for me this does feel a bit like pushing into the realm of diminishing returns (but barely)
There was definitely some system configuration stuff to troubleshoot to make this work properly (including obtaining the relevant dependencies and GNU Astronomy Tools) but nothing I couldn’t figure out or that Gemini / forums couldn’t assist with
It was really fun to develop this script and I will continue to use this pipeline despite the marginal (though definite) improvement in the final integration. If you know what you’re doing, AI can provide a huge value add to our workflows
Who is this for (in my opinion)?
People who like experimenting with workflows and different processes
People who like to tinker on the software side
People who live in high light pollution areas
People who live in areas that have poor seeing or infrequent clear nights
Live in a Bortle 5 that always has weird high altitude clouds? Is your air super humid? Are you lazy like me and start shooting targets at WAY to low of an altitude?
Who is this not for?
If you don’t like to tinker or troubleshoot
If you already produce incredibly high quality subs
You shoot from low light pollution areas, have great seeing / transparency, or both
Final Thoughts
This script / processing method is not some ultimate end game that everyone should strive for. It is an interesting methodology that works in some (many even) circumstances, but it will not turn backyard astronomy into JWST images. This process is not a background removal / gradient correction replacement, it is an additional step that improves the quality and ease of use of traditional BG removal methods.
At the very least it was a great experiment in creating custom scripts that can assist in the pre-processing of astronomical images. I encourage you to give it a try if this kind of thing is to your liking.
The attached scripts are generalized versions of the ones I developed for myself. They will almost certainly need modifying to suit your system and hardware configuration. I am posting these here for others to use and continue to develop as you want, but I did this project for myself and will not be maintaining public version of these scripts, nor will I be attempting to make these scripts Windows or Mac OS compatible, though I encourage others to give it a go!
GitHub links:
Pipeline:
Monitor:
📷 M57 - The Ring Nebula