Self-built gradient-removal pipeline: comparison vs PixInsight WBPP (Coma Cluster + Leo Triplet, identical data)

I also would love to have more insight into the workflow. The results on first look seem quite impressive! I have an image of the Leo Triplet as well where I just could not manage to get the tidal tail to show. I would be really interested in seeing what this method could do with my data.

Validating on an IFN rich target would be nice to see as well. Polaris is pretty rich with it and always accessible, though I know imaging Polaris can come with other challenges you may not want to deal with for this purpose.

Really interested in seeing how this develops!

The approach looks promising but your comparison images are not a good example in my opinion.

You should show:

• The original WBPP image.

• How your algorithm performs against other gradient removal tools like DBE, MSGC or Graxpert.

• A ground truth reference image for the background that shows any large scale low surface brightness features that should be preserved during gradient correction.

Curious to try as I have some image with remaining gradients I just can't get rid off, living in Bortle 8

@Victor Van Puyenbroeck, @andrea tasselli — you're both right, and thank you for the pushback. The WBPP-only comparison was not the fairest demonstration. WBPP doesn't do post-stack BG removal, so comparing against it shows the absence of any gradient tool, not the quality of the Abyss like algorithm against the established tools.

I'm currently running a controlled comparison study with identical pipeline up to and including StarAlignment + LocalNormalization, varying only the gradient removal step:

• Pipeline A: WBPP standard (no BG removal — current baseline)

• Pipeline B: WBPP + DBE (manual sample placement)

• Pipeline C: WBPP + GraXpert

• Pipeline D: WBPP + GC (GradientCorrection)

• Pipeline E: My pipeline applied post-stack (algorithm comparison)

• Pipeline F: My pipeline applied pre-stack (architectural comparison)

Same calibrated subs, same StarAlignment, same LocalNormalization, same ImageIntegration, same stretching — only the BG removal step differs. Follow-up post coming with quantitative comparison (MAD, low-frequency residual variance, autocorrelation at large scales) and side-by-side visuals across all 6 pipelines.

The "8-14× flatter" claim was specifically vs raw WBPP and I should have stated that more clearly. Against DBE/GraXpert/MGC the gap is smaller in raw MAD but likely very different in spatial residual structure — which is the perception-driving metric. I'll show both.

Thank you valuable critique.

Dirk Bausch · May 15, 2026 at 07:17 AM

@Vin — yes, very interested in your 14hr Virgo IFN data. That would be an outstanding test case for a target type where my pipeline could potentially struggle (faint extended LSB at scales near the cutoff). Could you share? Would include results in the follow-up post.

Ok I’ll DM you a dropbox link to 3 files - R, G & B masterfiles (the only thing that they’ve had done to them is calibration of the underlying lights w darks, flats, and dark flats - and then stacking - in APP).

I can’t understand the subtleties of the maths processes you describe above, but all I know is that if an improved version of gradient removal can be developed, that’s great! Will you also compare your approach to the PI Multiscale GC (although that seems to have big chunks of sky that are not covered by it yet sadly)?

No worries @Dirk Bausch - DM sent with dropbox links for 50 R files (sorry I don’t shoot L, only RGB separately). Fingers crossed your approach works well.

Brief follow-up addressing what came up in the thread.

## The proper comparison

@andrea tasselli was right: comparing against WBPP-only wasn't the right test. So I ran a controlled gradient injection — Urban LP + vignette, ~94 ADU spread across the integrated stack, 230-530 ADU per individual sub. This is hopefully representative of typical Bortle 7-8 LP+vignette conditions (most amateur observers).

Same 50 calibrated subs, four tools at default settings, integration through to master:

📷 l2_real_comparison.png

Visually clear: GraXpert, AutoDBE, and GC at default all reduce the gradient but leave visible residual structure. Pre-stack ABYSS produces a flat result.

Quantitatively (BG residual peak-to-peak on the integrated stack):

| Tool | residual | reduction |

| L2 BEFORE | 94.3 ADU | — |

| GC default | 39.8 ADU | 2.4× |

| AutoDBE default | 23.2 ADU | 4.1× |

| GraXpert default | 12.5 ADU | 7.5× |

| ABYSS pre-stack | 3.4 ADU | 27.4× |

Pre-stack achieves ~4× lower residual than the best post-stack default. On clean Bortle-5 data (Coma cluster from the original post) the gap is marginal — pre-stack matters specifically when the gradient is severe enough to stress post-stack assumptions.

---

## To specific commenters

@ValeryL — good suggestion on starless comparison. I'll do that for the IFN test (it's the right view for faint diffuse signal). For this gradient comparison the integrated stretched images already make the residual visible without starless, but for ICL/IFN preservation the starless view is more informative.

@Tony Gondola, @Jake Turner, @Joseph Biscoe IV — workflow is as described: per-sub à-trous starlet decomposition + NoiseChisel source masking, applied before integration; sits before LN/NSG, doesn't replace them. Uses open source tools.

---

## What I'm retracting from earlier in the thread

- "+0.7 to +1.2 mag deeper" — that was against WBPP without BG removal. Against actual BG-correction tools at clean conditions the gap is marginal (~0.07 mag).

- "8-14× flatter" — also against the no-BG baseline. Against GraXpert default at clean conditions it's ~16%.

The pre-stack architecture matters at stressed conditions, not universally. Proof of concept rather than pre-stack gradient removal is best.

---

## Where this doesn't help (and what's still untested)

- Hardware artifacts (focuser light leaks, internal reflections) — fix the hardware first

- Cellular/discontinuous gradient patterns — pre-stack uses smooth multi-scale priors and will fail

- Clean Bortle-5 conditions — no meaningful advantage over otgher gradinet removal tools at default

- Extended-nebula targets where most of the frame is signal (large emission nebulae filling the field, wide-FOV nebulosity, dense IFN-rich regions): the source-masking step requires enough sky-only pixels to fit a meaningful BG model. If the source mask covers >70-80% of the frame, the fit becomes unreliable. ABYSS hasn't been tested on this regime — open question

- Limited dataset coverage so far: N=2 (Coma cluster + controlled synthetic injection). Generalization to other targets, filters, sky conditions, RGB Bayer-pattern data, and multi-night campaigns with varying conditions remains open

Multi-scale wavelet BG modeling itself isn't novel (Bertin+1996, Borlaff+2019, similar in class to PI's GC). What's specific here is two things:

1. Pipeline position: per-sub, pre-stack. Adam Block has discussed in his educational content the principle that fixing issues in the data pre-stack can make data useable. Pre-stack gradient correction may be a natural extension. This pre-stack slot in WBPP isn't currently filled by post-stack tools, so the architectural distinction is real even though the algorithm class is shared.

2. Mathematical reproducibility: ABYSS is a deterministic analytical pipeline (à-trous wavelet decomposition + sigma-clipped sky fit + NoiseChisel masking). Same input → same output, every time. The algorithm steps are mathematically specified and auditable end-to-end. Useful in any context where the BG correction needs to be traceable. Complements rather than replaces tools optimized for ease-of-use or general-purpose convenience.

This could in principle be implemented natively in PI as a complement to GC/DBE/LN/NSG before the integration stage.

Clear skies,

Dirk

Wei-Hao Wang · May 16, 2026, 08:12 AM

Dirk Bausch · May 16, 2026 at 07:04 AM

📷 l2_real_comparison.png

I never imaged in Bortle 7-8. For those who did, please comment on whether this kind of weird background pattern is really typical.

My general comment is that so far all your example images are fields of small galaxies. This is not very challenging. All the other tools you tested were designed to tackle much more challenging cases: fields with large nebulas. For example, your 2025 top pick, NGC6888 (nice image BTW), could be a good target for testing. I am very curious what you will get if you dig out the pre-gradient removed image of that and run your new algorithm.

The most fundamental issue in gradient removal is to separate structures that belong to real celestial objects and that belong to gradient. The former should be kept while the latter should be removed. Fields of small galaxies are easy because the galaxies are much smaller than the characteristic size scale of the background gradient. In other words, the galaxies can be well above the S/N threshold within a small area where gradient is almost negligible, and therefore they can be easily masked out. That’s why most extragalactic surveys don’t even need the method described by Borlaff et al. The true challenge comes when your targets and background gradient all share similar characteristic sizes and brightness and when there is little source-free background area in the image to construct a robust gradient model. All the other tools you tested were actually geared toward solving such challenging cases (with different approaches). Based on the example above, your tool seems to be able to handle the small-scale structure in the complex background pattern (I even hesitate to call it “gradient”) and remove it cleanly. This naturally leads to the question of what if such smaller-scale structures are actually nebulas? Is you tool smart enough to leave them and not removing them? This is why I suggest to use your own NGC6888 as a test target.

These are much stronger gradients than what I experience in Bortle 8-9. I have less trouble with emissions nebula thanks to narrowband imaging so this is a very interesting endeavour still imo.

@Dirk Bausch I can provide Bortle 8-9 mono or osc raws with calibration frames from a range of targets if needed.

Wei-Hao,

your framing was exactly right. I took NGC 6888 literally — Baader 3.5 nm UNB, 87×300s, ~7.3h. Honestly? On targets like this I had usually skipped post-stack gradient correction anyway, because the results always felt unsatisfying. Your comment forced me to ask why methodically.

Two things came out of it.

First, the field really is BG-free in Hα. Stefan Ziegenbalg's Northern Sky Narrowband Survey shows the entire FOV filled with diffuse Cygnus-X filaments — there is no spatial region that qualifies as "clean sky" for fitting a complex BG model.

This is the practical version of your "free background area" point. Pro Hα surveys at this depth restrict themselves to a low-order plane fit at most for exactly this reason; anything higher-order catches real diffuse emission that nothing in the data can distinguish from gradient.

Second, there's a structural lower bound that any post-stack method hits: σ_residual is bounded below by the per-sub gradient variance. Pre-stack methods aren't bounded by this because at the per-sub level, the diffuse signal (~1-3 ADU) sits below photon noise (~10 ADU) while the gradient (5-30 ADU) sits above — they're separable per-sub but not after stacking. This isn't an ABYSS-specific point; it's true for any pre-stack architecture (pixel-space, catalog-based, etc.).

Tested it directly with synthetic gradient injection at three Bortle-realistic levels (1.5%, 2.4%, 8.2% of sky). ABYSS recovers σ_sky to noise-floor at all three — so the tool does detect and remove gradients when present, it's not just "leaves everything alone".

On real NGC 6888 data:

| Method | σ_sky | r vs IPHAS (nebula-only) | F_diffuse vs prelim |

|---|---|---|---|

| prelim (no BG) | 5.14 | 0.43 | — |

| GraXpert (default) | 4.22 | 0.34 | +10.5% * |

| AutoDBE | 4.24 | 0.35 | −0.6% |

| GC (PI) | 4.27 | 0.35 | +6.4% |

| MARS / MGC | 3.90 | 0.35 | +1.1% |

| ABYSS pre-stack (MFB-p2) | 3.82 | 0.44 | −7.9% |

(* GraXpert "gain" is a sky-pedestal-shift artifact, not real signal.)

What you predicted shows up in r vs IPHAS: self-contained post-stack tools drop morphology correlation from 0.43 → 0.34 because they treat the diffuse Cygnus filaments as gradient. MARS escapes this by bringing in external reference data — and is the standout post-stack performer in our test, exactly as Vicent Peris's architectural choice predicts. ABYSS escapes via the pre-stack architecture argument.

The open question I'd put back to you and the thread:

> On fields like NGC 6888 — no free BG region, gradient and diffuse signal sharing scales after stacking — is self-contained in-frame post-stack gradient removal a well-defined problem at all? Or is it information-theoretically restricted to either (a) external reference data, (b) pre-stack methods, or (c) accepting a low-order plane fit only?

If (c) is the honest answer, large parts of amateur NB post-processing are working on something that doesn't have a clean in-frame solution — and the workflow advice should reflect that.

Thanks for the original challenge!

Dirk

Hi Dirk,

In your NGC6888 test, did you use the Ha image or R broad-band? I think it will be more challenging in R than in Ha, as the background brightness (and therefore the gradient) will be much higher in R.

I really don’t feel there is a perfect answer. NGC6888 is one example to this challenge. Another example may be a 200mm FoV toward anywhere near the Milky Way plane especially near the center of the summer Milky Way. There, as long as the FoV is not pointed exactly at the Milky Way plane, there will be a gradient caused by the Milky Way. This gradient can be easily mixed and confused with the sky gradient. There is really no way to exactly separate the two gradients without relying on external information. The famous Dragonfly Array project by Yale also developed their algorithm to preserve large-scale IFN (in professional astronomy, this is called “cirrus” instead) reflection by referencing to far-IR and radio neutral hydrogen maps. There is just no other way round. They do reasonably well by using far-IR data to guide the algorithm on where the cirrus (IFN) may be. So ultimately, one will need some external guide.

Personally, I have been taking the MARS approach for almost two decades. I took very wide-field image first, to provide the “external information.” If the FoV is wide enough, wider than the target and wide enough to include many nebula-free regions all over the FoV, then a good gradient removal can be done on such wide-field image using existing tools (like DBE). Then such a gradient-removed image can serve as a reference for the narrow-field image. This is essentially what the MARS project tries to do, and I can’t think of a better method.

But, back to the images with many tiny galaxies, the Borlaff et al. method can actually work. If the goal is to preserve IFN while removing gradient, there is hope to actually achieve such a goal as long as the characteristic scale of IFN is smaller than the scale of gradient. Some parameter tuning may be needed for a different image, but fundamentally this is achievable as long as the two scales are different. All I want to say here and in my previous reply is that this method may run into trouble if the target scales are larger and comparable to the gradient size scale.

Cheers,
Wei-Hao

@Georg N. Nyman — completely fair point, and exactly the same critique that @andrea tasselli raised earlier in the thread. I edited the OP today with an UPDATE block at the very top retracting the WBPP-baseline claims and pointing to the proper comparisons (controlled gradient injection test against GraXpert / AutoDBE / GC / MARS at default settings, further down in the thread, plus an NGC 6888 stress test suggested by @Wei-Hao Wang).

Short version: pre-stack's advantage vs proper BG-removal tools is real at stressed conditions and dense fields, marginal at clean ones. Apologies for the framing in the original post — the UPDATE block at the top now reflects the corrected scope. Thanks for engaging.