What Could Be Causing This Random Noise All Of A Sudden?

Arun HFabian Butkovichandrea tasselliScott Badger
27 replies1.3k views
Fabian Butkovich avatar
Recently when processing data on several deep sky targets I've been imaging within the course of the last month, I've encountered seemingly unexplainable phenomenon with the stacked data, I am noticing very high levels of noise AFTER stretching luminance levels which were never as apparent in previous data.

I haven't changed anything about my imaging setup, neither physically nor exposure wise. On all my DSO imagining sessions I use 120 second long subs @ f/8 on my Sigma 150mm-600mm f/5.6-6.3 lens, the camera is an unmodified Canon EOS M50 Mark II. 

I usually try to aim my histogram to be centered on the 1/3 point on the graph, but obviously that will change with sky brightness and transparency. 

This is an Arcsinh stretched (to reduce star bloat) stack of M33 I acquired last month across several nights:



There is noise but very minimal and the sky looks mostly flat and smooth, I should note this stack includes flat calibration frames but I still had to remove gradient. 

Here is an example of the noise I've been encountering recently, this is the same Arcsinh stretch applied to a stack of M45 and M42:





I have tried stacking both with and without flat frames so see if maybe I have bad flat frames but it doesn't make a difference. I have also tried stacking with darks and they don't seem to make a difference either. 

If anyone would like to try processing to see if maybe I am just over-stretching my data or doing something else wrong, here is the link to the stacked TIFs:

M45 Stacked TIF
M42 Stacked TIF

I just don't understand what could cause the data to get to where I can't even do a simple stretch without the background looking so ugly.
Helpful Engaging
andrea tasselli avatar
The reason is PRNU or otherwise known as not having done a proper flat-fielding (or none at all as it looks likely from the pictures you posted) of the raws (plus potentially other issues such as not removing darks but is a lesser issue). Applying proper flat field with the proper technique is VITAL in data reduction as applied in AP.
David Koslicki avatar
Agreed that it's likely a calibration issue: the dust motes in the second shot in particular lead me to believe this. The fixed pattern noise could also be a problem with darks or biases. Is the "autorotate images" setting on the DSLR disabled? If not, there's a potential this could cause calibration issues, depending on what program you use to stack your images. 

Also, what program are you using to process your images? I've had a few images turn out like this when I was using DSS & photoshop, but the problems went away with a combo of better calibration on more sophisticated image processing software (that can do thing like background extraction, LP removal and the like). If you're interested in seeing if stacking/processing is contributing to it, I'd be happy to give a go at processing your raw subs + calibration frames.
Helpful Respectful Engaging Supportive
Fabian Butkovich avatar
David Koslicki:
Agreed that it's likely a calibration issue: the dust motes in the second shot in particular lead me to believe this. The fixed pattern noise could also be a problem with darks or biases. Is the "autorotate images" setting on the DSLR disabled? If not, there's a potential this could cause calibration issues, depending on what program you use to stack your images. 

Also, what program are you using to process your images? I've had a few images turn out like this when I was using DSS & photoshop, but the problems went away with a combo of better calibration on more sophisticated image processing software (that can do thing like background extraction, LP removal and the like). If you're interested in seeing if stacking/processing is contributing to it, I'd be happy to give a go at processing your raw subs + calibration frames.

I hadn't thought about the rotation of the frames, I wouldn't think this would matter as DSS obviously rotates them when stacking; there could be a handful of rotated frames after the meridian flips. To be honest I hadn't stacked any darks with either data set only because while I have collected multiple sessions of darks recently in the morning after my imaging session are done, I'm not entirely sure how I would match them temperature-wise with my light frames. 

This is what my stretched master flat looks like and it definitely is evident that the same dust motes appear in my stretched version of M42.



Please feel free to stack and process the data yourself! I'm very intrigued to see if you can produce a different result. 

https://drive.google.com/drive/folders/1Hghv61Dc_iaERIC2OAp3zSLGDxiF0G9m?usp=share_link

My workflow is that first I'll import the raw files into Adobe Camera Raw and set the white balance to align all histogram channels, then I'll apply lens profile correction for my lens and last I'll modify the exposure to center the histogram peak around the 1/3 point if not already (not sure if this accomplishes anything). After making all these adjustments in camera raw I'll then export to .DNG into the "Lights" subfolder in my DSO directory and stack these in DSS.
Helpful Engaging
Jonny Bravo avatar
I see raw lights and flats… where are your biases / flat darks? You need to have them for your flats to properly work.
Concise
andrea tasselli avatar
Fabian Butkovich:
My workflow is that first I'll import the raw files into Adobe Camera Raw and set the white balance to align all histogram channels, then I'll apply lens profile correction for my lens and last I'll modify the exposure to center the histogram peak around the 1/3 point if not already (not sure if this accomplishes anything). After making all these adjustments in camera raw I'll then export to .DNG into the "Lights" subfolder in my DSO directory and stack these in DSS.


Don't do any of that. Just use the workflow provided with DSS using only the raw calibration and light frames.
Fabian Butkovich avatar
@Jonny Bravo I don't take these calibration frames, I was under the impression that flats and darks alone are good enough, and darks themselves are sometimes redundant if the stacked data doesn't have much sensor noise to begin with. I didn't know that flats depend on biases and flat darks.
Jonny Bravo avatar
Fabian Butkovich:
@Jonny Bravo I don't take these calibration frames, I was under the impression that flats and darks alone are good enough, and darks themselves are sometimes redundant if the stacked data doesn't have much sensor noise to begin with. I didn't know that flats depend on biases and flat darks.

Flats depend on either a bias or a flat dark. It's in the math. A simplified equation for image calibration is:

(light - dark) / (flat - bias)

A light has all of the signal you want, plus the dark current, bias, optical imperfections, vignetting, etc etc etc.
A dark has the dark current plus the bias
A flat has the dark current, the bias, the optical imperfections, vignetting, dust motes
A bias has the bias

So, looking at the equation, the numerator has subtracted the bias and dark from your light (since the dark contains both bias and dark). If you do not include the bias (or flat dark) in the denominator, then you're dividing by a flat that still has the bias. This will lead to incorrect calibration. You _need_ to get rid of the unwanted signal from that flat frame.

I won't get into the whole bias vs flat dark debate - it's been done ad nauseam .
Well Written Helpful Insightful Concise
Scott Badger avatar
Any chance you changed your ISO? Canons are notorious for a banding issue that with the 5D Mark IV I use to use was much more prominent at certain ISO settings. Going from memory, I think shooting at 800 was the worst, but higher like 1600 or 2500 wasn't nearly as bad. Later tonight I'll try to dig up some of those problem images to see how similar it was to what your seeing. Also, I agree with Andrea Tasselli to skip the pre-pre-processing. As I understand it, one of the benefits of astro specific cameras is that they truly pass raw data where DSLR's 'raw' isn't quite raw…..

Cheers,
Scott
Helpful Supportive
Arun H avatar
You really do need to subtract the bias(or flat dark) from the flat as Johnny has suggested is because the bias can be significant. A zero exposure time subframe on my 294MM, at Gain 120, has an ADU level of about 1600. If you are aiming for a flat ADU level of ~20,000, you are adding a nearly 10% error when you are calibrating -very simplistically, you are dividing by 20,000 when you should be dividing by 18,400. Subtracting either a bias or flat dark (which contains the bias) assures that you are not introducing this error and calibrating correctly.
Helpful Respectful Concise
Fabian Butkovich avatar
Scott Badger:
Any chance you changed your ISO? Canons are notorious for a banding issue that with the 5D Mark IV I use to use was much more prominent at certain ISO settings. Going from memory, I think shooting at 800 was the worst, but higher like 1600 or 2500 wasn't nearly as bad. Later tonight I'll try to dig up some of those problem images to see how similar it was to what your seeing. Also, I agree with Andrea Tasselli to skip the pre-pre-processing. As I understand it, one of the benefits of astro specific cameras is that they truly pass raw data where DSLR's 'raw' isn't quite raw.....

Cheers,
Scott

My ISO is always 800 as this is the sweet spot between read noise and dynamic range for my specific camera. Originally I would shoot at 100, but someone had pointed out to me that the read noise is at it's peak at IS0 100 for the EOS M50. I just don't understand why I would notice this banding issue all of a sudden even though like I mentioned everything had stayed the same between imaging sessions.
Scott Badger avatar
Again from memory, but I think I didn't have the banding issue at first using 800, but don't have any guess as to why it wouldn't be an issue right from the start.

FWIW, I just found an old Astrobin thread ('DSLR Exposure Time') where I asked about the banding issue and at that time, reported it as 'intermittent' and Arun, maybe you remember replying:

"Scott - I think the reason that banding is ISO dependent has to do with the way Canon sensors are constructed. There are two components to read noise - the noise from the sensor itself and the noise introduced during analog to digital conversion. The latter is quite significant. The higher the ISO, the more the signal from the sensor is amplified before the A-D conversion is done, and hence the lower the contribution from the A-D conversion noise. Generally for Canon sensors ISO1600 and up seem to be sweet spots, though you lose one stop of DR for every doubling of ISO beyond 1600 for older Canon cameras and 400 for the Mark IV.
I have used and continue to use on occasion, a Canon 5D Mark IV (unmodified). I would always get one or two very strong bands, regardless of ISO. Using darks without biases helps get rid of them."

Cheers,
Scott
Helpful
Arun H avatar
Scott - this is accurate. The older Canon sensors did the analog to digital conversion off sensor, which introduces more electronic noise. The banding comes in because different columns of data use different pathways before the A/D conversion is done, hence the introduced noise has regularity. Increasing ISO amplifies the analog signal before the noise is introduced, hence suppresses the banding. Sony sensors, which are what are used in most astro cameras these days don't have that issue. Yes, that was my experience with the 5D Mark IV. As to darks without biases on that camera - not wanting to derail this discussion, but I am deeply distrustful of biases on CMOS cameras because they can have subleties that cause them to behave very differently at short exposure times (where you take your true biases) versus long exposure times (for lights and flats). My preference has always been, for that reason, to calibrate both flats and lights with darks of the same exposure length and, where practical, temperature. As long as the sensor behaves in a repeatable fashion, there is little downside to calibrating just with darks rather than worrying about whether or not the bias is accurate.
Helpful Insightful Engaging
Arun H avatar
For the original poster - if using PixInsight, it has, or used to have, a Canon Banding Reduction script, since this banding problem was so prevalent in Canon cameras. It would be worth trying to use it to see if it eliminates the banding. The script needs the bands to be horizontal where they are vertical here, but that is easily addressed by rotating the image by 90 degrees before applying the script.
Well Written Helpful
Scott Badger avatar
Found one of my problem images in a CN thread and now that I see it again, the banding problem I had looks quite different from what you’re getting…..sorry if I raised a red herring! FWIW, I stated in CN thread that the banding wasn’t always a problem. Maybe the banding plus a calibration issue?
andrea tasselli avatar
Arun H:
For the original poster - if using PixInsight, it has, or used to have, a Canon Banding Reduction script, since this banding problem was so prevalent in Canon cameras. It would be worth trying to use it to see if it eliminates the banding. The script needs the bands to be horizontal where they are vertical here, but that is easily addressed by rotating the image by 90 degrees before applying the script.

Is not a banding issue here, at all. It is an issue of bad data reduction. I had a check at their data and for starters the flats don't match the lights. No bias/dark flats/darks. The average value of the flats is way less than recommended and so on.
Helpful Concise
Fabian Butkovich avatar
andrea tasselli:
Arun H:
For the original poster - if using PixInsight, it has, or used to have, a Canon Banding Reduction script, since this banding problem was so prevalent in Canon cameras. It would be worth trying to use it to see if it eliminates the banding. The script needs the bands to be horizontal where they are vertical here, but that is easily addressed by rotating the image by 90 degrees before applying the script.

Is not a banding issue here, at all. It is an issue of bad data reduction. I had a check at their data and for starters the flats don't match the lights. No bias/dark flats/darks. The average value of the flats is way less than recommended and so on.

I thought that flats can be applied from different sessions to any lights? as long as the focal length and focus stays the same which in my case hasn't changed hardly between data.
andrea tasselli avatar
Fabian Butkovich:
I thought that flats can be applied from different sessions to any lights? as long as the focal length and focus stays the same which in my case hasn't changed hardly between data.


For DSLRs this is very much *NOT* recommended as the dust motes tends to move quite a bit from time to time when the camera is powered down. The problem with your data set is that the flats should really be, on average, having a mean value between 1/3 to 1/2 of the total dynamic range at the same ISO of the lights and from what I can see they are nowhere near.

As many have said before me, getting the dark-flats or biases and subtracting their master file from the flat lights is key to a proper data reduction otherwise all bets are off that you will end up with a properly flattened stacked light, which I suppose is one of the problems you see in your images (and this includes the M33 shot too).
Helpful
Arun H avatar
Flats can indeed be used between sessions. You only need to worry about whether dust motes have moved.

As to the banding - regardless of cause, I think  it is worth the two minutes to apply the Canon Banding Reduction script. The cause does not matter - the way the script works is to calculate a median background value and add or subtract the difference from the median to each line. It will work on any regular horizontal bands and it did remove the type of green band Scott shared in his image in my case.
Well Written Helpful Insightful
Andy Wray avatar
I would agree that it looks like a flast/dark flats issue.

FWIW:  you are also not cropping your stacked images before processing them to get rid of stacking artefacts.  That, in itself, will also make your post-processing worse..

Here's the best I could do with your M45 image.  I'm sure others could do much better.:

Stuff I did:  

* Croppoed out the stacking artefacts
* Extracted stars with StarXterminator
* Used automatic backgroud extraction to get rid of the worst of the background colours
* Used heavy NoiseXterminator on the starless image (probably too much)
* Applied a range mask on the starless image so that I could then use curves to dial out the saturation of the background and darken it (probably too much)
* Had to apply convolution on the stars to get rid of the donuts
* used PixelMath to recombine starless and stars

This was a very quick and dirty attempt to get something out of a very noisy jpeg image, however with careful post-processing on the raw data you may be able to get something a lot better.
Helpful Supportive
Arun H avatar
Here is Canon banding reduction applied to your M45 JPG.

As I expected, banding is significantly reduced. Applying it to the original TIFF and processing should help.

This is not to take away from making sure your flats are well taken and calibrated, etc. That absolutely should also be done.  And I will agree with Andrea on the point that, if there is any question about dust moving, you should take flats after each session. 



Here is your M42 image, which also shows banding improvement. You have dust donuts which suggests that the flats are not properly correcting, which may be a different issue than the banding. 

Well Written Helpful Insightful
David Koslicki avatar
@Fabian Butkovich 
Hi Fabian, 

I gave a quick go at processing your subs, and wanted to share my observations: 

1.     Given this is a multi-night imaging session, your framing is rather bang on! I recall when I was using a DSLR, my stacking artifacts (from non-perfect framing) were a lot more than yours. So how ever you are going about the plate solving etc. keep at that.

2.     There were a fair number of images that had guiding errors, wind, vibrations, or the like that caused the stars to smear. I blinked through these, but instead of manually finding the best frames, I ended up just using AstroPixel Processor’s (APP) quality feature to select the best 50%.

3.     As has been said frequently here, the flats and darks/biases are a must. APP didn’t like the flats at all, so this stack is just of your lights. Note that it’s more important that you use darks, rather than omit that step because the temperature might not be “exactly correct.” Close enough is better than not at all, in my experience at least. Same with lights: better to get most of the motes than none of them.

4.     You may consider shorter exposure times. Even without any stretching at all, the core of Orion is already blown out/saturated. Shorter exposures would allow you to recover detail in the core, and wouldn’t appreciably sacrifice detail elsewhere.

5.     As mentioned here before too, you don’t want to fiddle with/change your subs at all before stacking them. Differences in illumination issues (eg. transient light pollution) and the like are post-processed out or handled by the outlier rejection of stacking. Trying to fix them pre-stack plays havoc on the stats/algorithms of the stacking process. 

My basic processing steps were:

a) use APP to register and normalize (advanced normalization) the RAW *.cr3 files

b) blink through to get an idea of the % of good subs,

c) stack with a quality cutoff of 50%,

d) use the light pollution tool (dynamic background extraction/neutralization/calibration all in one),

e) calibrated star colors based on an adaptive black body & extinction model,

f) removed the stars with Starnet2,

g) ran Topaz Labs denoise AI on the starless image (severe noise, all settings set to 0/100 except noise removal 4/100, then set opacity to ~80% in PS),

h) used photoshop to add back in the stars and do a bit of cropping, rotating, sharpening, histogram fiddling, etc. Not much could be done to save the blown out stars, hence the suggestion for shorter exposure time. 

FWIW, I found my images improved significantly when moving from DSS to APP. I don’t know if it’s more sophisticated registering/normalizing/stacking algorithms, or what, but you might consider playing around with different software. APP has “multi-session imaging” so you can match your darks & flats to each night (i.e. take flats before imaging, darks just before bed, and calibrate each night’s subs with the corresponding calibration frames).
Note: I have yet to “bite the bullet” and learn PixInsight, so I can’t say much on that front. 

Here’s the image: https://www.dropbox.com/s/b29lw6q1hzdnxan/For_Fabian-RGB-session_1-St_starless.jpg?dl=0
 And here’s the PSD file: https://www.dropbox.com/s/9mrdapgsjs963qn/For_Fabian_guy-RGB-session_1-St_starless.psb?dl=0 

tl;dr: the lights are rather good. You’ll likely be pleased with proper calibration, reduction in exposure time (and/or ISO), and alternate processing.
Well Written Helpful Insightful Respectful Engaging Supportive
Fabian Butkovich avatar
@David Koslicki wow! I am quite impressed with how clean you managed to process my data. I actually had a go at re-stacking in DSS with just the raw frames and also included about 12 or so darks that were the same temperature as the average across all the light frames and flats. The noise was tolerable. I'm not familiar with APP but I've heard of it. I recently downloaded a free trial of PixInsight to see what the hype is about but it's definetely not something that's the most user friendly from first impressions. 

Respectful Supportive
Andy Wray avatar
Fabian Butkovich:
I recently downloaded a free trial of PixInsight to see what the hype is about but it's definetely not something that's the most user friendly from first impressions.


I thought that about Pixinsight at first, but I persevered for a few days and it ends up being very intuitive.  Try working with WBPP as that is the nearest thing to DSS and you will soon get used to it.
Supportive
Scott Badger avatar
FWIW, 'Mitch' has a great series of introductory tutorials on Youtube. Very detailed and though basic, the results were still pretty good. It was a good way to start learning the PI environment.

Cheers,
Scott
Related discussions
New to or Considering Astrophotography/astronomy?
I had posted this with a new image I’d uploaded as a change from the “typical” review of my processing steps/acquisition. A friend/commentor suggested I post in a forum, so this is the result. This is VERY subjective and based entirely on my experien...
Jul 31, 2025
Both posts appear to be incomplete, cutting off mid-sentence while discussing astrophotography processing issues.