How much is enough data? - Forums

Topic starter

Jun 17, 2020

Hey all, still learning the ropes but one thing I do know is that the more data you have the better (depending on the quality of course). The question is when is enough enough? I started out working on M81, which initially was around 2 hrs and now I am at the 14 hr mark. Same with M27 2 hrs, now 7hrs.

I plan to add more data but when should I stop? Will I really add more noise? Are there other factors or should I keep going.

Thoughts are most welcome.
Brad
Clear skies

Respectful Engaging

Steve Milne

Jun 17, 2020

Hi Brad. Great question. Get ready for a rubbish answer

The more subs you have, the more your Calibration and stacking software should be able to discern ‘real’ photons (signal) from random stuff (noise). The more noise you can eliminate in the stack, the less NR and other problems you’ll get as you get into the processing. But it does become a ‘diminishing returns’ thing.

I prefer to have many hours of data, so a typical image of mine would get into the 10-20 hours mark, and then some. Others prefer to capture more targets in the time they have available to them.

If you’re just starting out why not process the 14 hours of M81 that you have and compare it with the 2 hour version? It should look better. Then, if you’re up to it, get a bit more and see if that produces a further improvement. At some point you’ll start to work out the point at which you don’t feel you’re seeing any improvement. And that is your minimum.

I think more data almost always helps, so long as it’s in focus and so forth. Don’t shoot RGB data when there’s moon around, even if your target is a way away from the moon.

Steve

Well Written Helpful Insightful Respectful Engaging Supportive

Kevin Morefield

IOTD Staff

Jun 17, 2020

Bradley,

Agree with all Steve said. How much data to reach a given signal to noise ratio depends on how bright the object is, how fast your OTA is, how big your pixels are, how high the Quantum efficiency of your sensor is, how low your read noise is, and lastly, how much light pollution you are fighting.

But I wouldn’t suggest you try to put all of that into a formula. Realistically, after processing a few images you will get a feel for when more data is not adding enough improvement to the signal to noise ratio. With a fast system and a low read noise camera 5-7 hours can produce a decent image. I usually end up shooting more like 10-20 hours because that allows me to “push” a harder in processing to get more contrast and saturation without egregious noise. But I have automated the system so I’m not balancing sleep vs signal to noise!

I suggest you take a target you like and shoot a lot of data - whatever feels like a lot to you. Then process the stack with only one hour, then two, then four, etc and see what feels like the best answer for your goals.

Kevin

Helpful Engaging

Bradley Watson

Topic starter

Jun 17, 2020

Kevin and Steve, thanks for the feedback. I suppose I can’t go wrong with adding more data, it will be a good exercise to get a feel of when adding more does give anything.

I am using my Skywatcher Mak 127 right now (not tried my new APO refractor yet), and I have read that it’s a planet killer but not much use for anything else. I don’t really get the logic behind this, maybe I am missing something. If my guiding is up to scratch and meets the demands of a long focal length and I capture enough data to offset the slow speed (F12) why is this telescope no good for anything else. I quite like the images I have and that is with very little experience.

Anyway thanks for the input, I will add more data to my current projects…………. I have a busy summer if the weather holds out! This hobby is a marriage killer lol
clear skies

Respectful Engaging Supportive

Kevin Morefield

IOTD Staff

Jun 17, 2020

Bradley Watson:
If my guiding is up to scratch and meets the demands of a long focal length and I capture enough data to offset the slow speed (F12) why is this telescope no good for anything else.

Two basic thoughts: The first equipment you should use it what you already have, and if you like the pictures, keep at it!

But here are a few things about the Mak that might not perform as well as more imaging friendly scopes:

- The image scale of the Mak/Canon 70 combo is .56 arc seconds per pixel. That's pretty small and would need world class seeing and guiding to fully use. For average seeing an image scale of 1.0-1.5" would probably be better. Meaning, you would not lose any resolution in your final image, but with a larger pixel or shorter focal length you would gather a lot more photons per pixel. That means less exposure time needed and better signal to noise ratio
- The field of view is 0.86x.0.57 degrees. That's fine for small to medium sized objects like the Dumbell and all but the largest galaxies. A shorter focal length would let you go after wide field images too, again, without losing resolution.
- The slow speed of F12 will definitely require a lot more exposure time compared to a faster scope like an APO refractors. I mean like 3, 4, or 5 times as much. And remember, even though the image of the object is bigger on the Canon sensor, with a longer focal length, you likely won't be resolving any more detail with that longer FL.

Sounds like you already have the APO, so it should be pretty easy to fire that up and see how the sub-exposures compare to the Mak. One thing about the APO and the Canon though, you might need a field flattener to get sharp stars in the corner. That may not matter for objects like M81 since the FOV is going to be so much bigger in the first place.

Helpful Insightful Engaging

Steve Milne

Jun 18, 2020

Kevin Morefield:
I don’t really get the logic behind this, maybe I am missing something. If my guiding is up to scratch and meets the demands of a long focal length and I capture enough data to offset the slow speed (F12) ...

First off, if you're having fun that really is all that matters. When I started I couldn't quite believe that I was able to capture anything at all. However, I agree with Kevin: it's crucial, when making any astro-imaging purchase, to consider imaging scale. I find this site very helpful: http://astronomy.tools/calculators/field_of_view/ The EQ5 is a great mount. But it is unlikely to be able to 'match' the ~0.6" pp imaging scale that your scope/camera combo gives you. (That isn't any criticism of the EQ5 - mounts costing many multiples of the EQ5 wouldn't be able to do it either.) A more realistic imaging scale would be in the region of 1-2" pp. I don't know what refractor you have but the great Skywatcher ED80 (with 0.85 reducer/flattener), for example, would put you in the ~1.8" pp range. You clearly have the system set up quite well since your stars are not too bad in the images on your site. I suspect they might get a bit tighter with a shorter focal length set up though.

Feel free to drop me a PM if you want me to go into this in more detail.

Steve

Well Written Helpful Insightful Respectful Engaging Supportive

Bradley Watson

Topic starter

Jun 18, 2020

Thanks Kevin. It’s funny, I loved my Mak and then loathed it but as I have got better over the weeks it has grown on me. I did research into the resolution for camera and telescope relationship as well as guide scope before. My typical total RMS for guiding varies between 0.7 and 1.6 total RMS which means I am throwing away a lot of frames. I also think this is the reason why my stars look like pancakes.

This is the reason I bought the Vixen but enjoying the Mak now but am also waiting for the dovetail as I can see without mounting, the the balance will be far out. Also I think the ratio for my guiding and the Vixen will be better too. I am glad you mentioned the flattener as I have been wondering about that. Is it worth looking at a flattener/reducer? My vixen has 600mm fl for F7.5. Would I be compromising anything with a reducer? Is it required?

I really, I mean I really appreciate the advice. There is sooo much info out there, it can be difficult wading through the weeds for the right answers.

Cant wait for my dovetail now!

Bogdan Borz

Jun 18, 2020 · Edited: Jun 18, 2020

Bradley Watson:
Hey all, still learning the ropes but one thing I do know is that the more data you have the better (depending on the quality of course). The question is when is enough enough? I started out working on M81, which initially was around 2 hrs and now I am at the 14 hr mark. Same with M27 2 hrs, now 7hrs.I plan to add more data but when should I stop? Will I really add more noise? Are there other factors or should I keep going.

Thoughts are most welcome.
Brad
Clear skies

Hi Bradley,

Steve and Kevin already gave some good points. I am sometimes asking myself the same question. However there is no clear cut answer for all targets, since all targets have different magnitudes.

As Steve pointed out, the main issue is SNR (signal to noise ratio). What you basically need to take into account is that noise is inversely proportional to the square root of the number of subs. So 59 subs will reduce the noise by a factor of 7 for example. Since we usually all have pretty noisy individual subs, you have to shoot some in order to increase the SNR.

The total integration time will depend on a number of factors: the magnitude of the target, the sky magnitude (your background brightness, which can be measured), the fastness of your system (the lower the F ratio the better), the read noise (each time you read a frame you add read noise). This can explain why when shooting from light polluted areas you're going to have to sometimes double the exposure time and the number of frames in order to obtain the same result as someone from a dark site. The dark site has much lower sky brightness to the SNR automatically increases. Craig Stark made a nice presentation available on youtube. All this stuff can be entered into a formula to give some idea when to stop. Sharp Cap pro has for example an utility for measuring sky brightness and camera parameters and give you an idea of the optimal sub length. When you keep adding hours that does not mean that the benefit stays constant: it will decrease asymptotically, adding 5 hours to arrive at 10h is much more spectacular in SNR than going from 20 to 25. So choose a fast system a flattener reducer brings you at F 5.6 from F7 with your apo for ex, look how bright the target is and where do you live. Shooting M42 is not the same as the Cave Nebula (I put 30 h into the cave without a spectacular success). 10-20h should do for a good start (I am very precise I know, I am not taking risks : ) ).

Clear skies,

Bogdan

Helpful Insightful Respectful

Bradley Watson

Topic starter

Jun 18, 2020

Hi Bogdan,

Thanks for the advice. A lot to take in

I think I will try to see where my limit is with the subjects I am working on now and then switch to my Refractor and see what I can learn in terms of differences between my Mak and my refractor.

I’ll take a look at Sharpcap Pro too, about to buy PixInsight

If I have shorter subs, the SNR is lower right, so with more subs I am introducing more noise. e.g. if I shoot a 5 min sub vs 5 1min subs, is the noise more or less? I know there are other variables but I feel like with my Mak with longer exposures I am introducing less noise but increasing signal. Challenge then becomes guiding and it’s painful throwing away 5min subs! Right now I think adding more data with an F12 scope can only be a benefit.

I will look at the formulas, everything helps right now

Thanks for your advice
I like your photos, you have a great library

Kevin Morefield

IOTD Staff

Jun 18, 2020

The most noise is introduced via the read noise. But more exposures will always improve SNR since you are introducing more signal than read noise with each additional sub. But for a given total exposure time, fewer subs is better than more subs. Caveat: some cameras, DSLR in particular, aren’t great with longer exposures. So you have to find a balance.

Helpful Insightful Concise

Diego Cartes Saavedra

Jun 18, 2020

Hello all,

I am very interested in this topic, because sometimes i don't know when to stop capturing data.

At least in my case, when i do narrowband, i prefer to capture a minimum of 10 hours of each filter. That's about one or two nights of capture per filter. But it's only a decision based of experience.

Regards,

Diego.

Bogdan Borz

Jun 18, 2020 · Edited: Jun 18, 2020

Bradley Watson:
If I have shorter subs, the SNR is lower right, so with more subs I am introducing more noise. e.g. if I shoot a 5 min sub vs 5 1min subs, is the noise more or less? I know there are other variables but I feel like with my Mak with longer exposures I am introducing less noise but increasing signal. Challenge then becomes guiding and it’s painful throwing away 5min subs! Right now I think adding more data with an F12 scope can only be a benefit.

SNR=So/N=So / SQRT (So+Ss+Sd+R2)

Here is the formula for SNR and here you can see the factors affecting the ratio (So is Signal object, Ss for sky, Sd dark and R is read noise).

An important aspect is this shot noise cannot be subtracted since it is random, this is not the dark current subtracted when doing darks. As you can see what matters is not only the absolute value of each contributor to noise, but their relative value. So if you are shooting under dark skies you're really going to be concerned by read noise. If you are shooting under LP skies like me (you are also in the same scenario I see), read noise is going to be much smaller than sky noise due to LP. In an ideal scenario shooting one 10 min sub vs 10 subs of 1 min would be equivalent. If you shoot a 10 min RGB sub under Bortle 7 skies (even with the LP filter) your object is going to be drowned in the sky signal. You cannot just increase the sub duration as you please since the sky (really) is the limit. The other aspect is the read noise with multiple subs. So yes, using 1 min subs will add more read noise. The question is not if the read noise increases, but what is the best ratio[b] [/b]compared to the sky noise.

As a practical example I used 30s and 15s L subs for my M13 photo. Pretty low gain 0.5 e-/ADU (under unity which is 1 e-/ADU). So I increased the read noise considerably. My signal for this bright object is high (So is high) and my Ss dominates (Bortle 7/ 8 ). Well, the background for this image was one the cleanest I've ever had, I used 192 subs which decreased the noise 14 times. But this scenario does not apply in your case, since your object, a planetary nebula is not as bright as a globular cluster.

By using narrowband filters, you will increase the SNR since you will decrease the Ss (the LP from the sky background is considerably reduced when you shoot through a 3nm passage vs 300 nm passage for L.

The duration of your sub length should be dictated by the ratio object magnitude vs sky magnitude under LP. Second aspect is your F ratio. Shooting with a Mak is not adapted to deep sky, it collects few photons. Each F stop will double the quantity you get on the camera chip. So go with the refractor at F7 or use a reducer to get it at F5.6 (0.8x reducer). Much better than the Mak (but less aperture and resolution of course). One aspect that people usually misunderstand is that by using a reducer you are not going to shoot the same object faster. The reducer will increase your field of view and more light will hit the chip, but from a larger field. Your object will generate just as much photons on the chip as before (it became smaller in the fov which compensates the increase in speed of your system).

I hope I was clear enough, this topic touches a lot of aspects so it is easy to be convoluted in what you're trying to express.

CS
Bogdan

Helpful

Alan Brunelle

Jun 18, 2020

I am also new to this endeavor and really appreciate all the comments above. I would like to add my thoughts since so many of the comments point to the very valuable issues/limitations to equipment, such as the OTA capability and the match to camera, SNR, etc. I think that these are crucial regarding what the gear package is capable of achieving (not to ignore seeing constraints as well). But given my very cloudy skies, I have taken the "opportunity" to reprocessing much of my old data and I am learning work flow in PI and other associated imaging processing software. What I am learning (and also seeing on this site and others) is that processing is a critical co-component to getting what can be gotten from the data. After all, most processing software deals heavily in noise correction, noise cancellation, image sharpening and the like, not to mention contrast enhancement. I have found that it is very easy to over work an image and thereby create structure that is not there or is improperly presented just as easily as overworking and under working can miss or destroy lots of structure. Of course, I want to get the absolute most out of whatever data I generate, whether 30 min or 30 hours. Other than the SNR considerations, as I work the images, I feel that more subs offer more "room" to work with. And as others have said, the best way I am learning this is by doing with identical data sets. And using identical data sets and processing less of the subs. Because image details and structure "reveal" themselves often only after processing, I have used others' images to see if I too am creating that which is simply not there. In fact Hubble photos are so detailed that if one exists for an object being worked on they can help guide, so long as one realizes that our equipment's limited resolution could never generate the sharpness that Hubble achieves. And not get depressed that much structure Hubble presents is not going to be seen by our instruments. (I sure hope that the Hubble image processors do not over work their data!) I do not intend to try to simply copy what others (and Hubble) have done, only as a guide that my hand is not too heavy with PI as I learn and that I get to see what is real. What I find a bit odd though, is that I often see images done with umpteen hours of data generated by f2 telescopes. As I got into this, I expressly went to the fast telescope precisely because I would not necessarily need to go so long! If the point of using an f2 is speed, then why use the same number of hours in exposure with the f2 as the f8? I do believe that there is a tendency to ignore just what the gains from using a fast scope means and conversely, fail to minimize the diminishing returns by going plainly too long. Since I am an old fart, I feel that putting too much collection time into a few images, when the product will not be noticeably better, means that I get to see less targets. I can care less of bragging rights as to how many hours of collection I have sweated over. Finally, I am learning that my more profound mistakes have less to do with using too few subs, but by using an inappropriate sub exposure time. And that goes for too long and too short.

Bradley Watson

Topic starter

Jun 20, 2020

Hi Bogdan,
Thanks for your comments. I had to read this a couple of times.

You have actually answered a big question I have had about my Mak. I think every stop down doubling light capture ability is pretty significant.

I think the key thing is to identify my limiting factor and decrease that, that would actually reduce the time required to get the same result. With my Mak with its F12 ratio the point where I get diminishing returns will be farther out right? For M81, I had 5 min subs in a 7/8 bottle sky, basically you are saying I would benefit hugely by reducing the duration because of the light pollution? I think I need to try my setup away from the city and see what the results are like, my SNR theoretically should be much better if I can get to a bottle 3/4 sky thus reducing integration time.

This is a big balancing act.

Bogdan Borz

Jun 20, 2020

Hi Brad,

Well in your case the benefit would be shooting under darker skies (this is kind of universal : )). If you are using a LP filter with a slow F12 Mak 5min can be ok (but again all this has to be judged on a case by case basis). I don't think you got too much LP / Signal at F12 (I shot 3 min with no LP filter at F4). But in order to increase signal with an F12 you will have to increase sub length and then the problem of guiding comes up. Your Mak has a long focal length and you have to calculate the sampling given the pixel size of your camera. Anyway, you cannot guide your Mak with a 1500mm focal length with the 50mm scope (it's too small). Plus you could have flexion problems. Off axis guiding is usually recommended for systems with this focal length. So you cannot increase sub length because of guiding, even if more could be necessary for signal increase. OAG with a DSLR is very exotic and I never heard of it (I don't even know if you have OAG with T2 adaptors). The solution if you want to keep your configuration is to use short enough subs to keep the stars round while still obtaining enough SNR (so bright targets like planetary nebulae or clusters + low sky signal ). And you have to increase total integration time.

Your Dumbell looks pretty good taking into account all these conditions. The bottom line is that the combo you actually use is a headache for shooting deep sky. If you could use a small refractor instead, it would really be a lot easier to obtain better pictures. The Mak is really good for lunar images.

And thanx for visiting my gallery ; ).

Helpful

Giovanni Paglioli

IOTD Staff

Jun 21, 2020 · Edited: Jun 21, 2020

Hi! I underestand the point of Your question and, many has given You excellent answers but I would like to extend the concepts a little more if I can. In terms of scientific use of the datas, there are rules You can follow that are based on the principle of "confidence" of the datas. It means that tha grade of "certain Vs. uncertain" (SNR) can be established before You acquire any image, this is based on measurements that are already been taken. First of all is the Photometry of the intended subject than the characterization of the system including the sky-noise component. Photometry is a measurement of the luminous flux that falls on to a given angle in time, that means You can know in advice how many photons per second are falling onto a pixel I.E. Charaterizing a system on the other hands takes in advice any loss and errors (including Q.E., Vs. ferquency of the spectrum, electronic, quantization errors, environmental conditions etc.) so You can know how much the "unwanted signal still present" contribute to an image. That said You can establish that a certain amount of exoposures on a 15th mag. subject will have a certain percentage of noise that is not possible to eliminate but can be measured. In this way You can establish how much exposure You need to have, i.e., a confidence grade of 3 which means having three times more signal than the uncertain from the subject. Is it also clear that this kind of improvement is not linear but tends anyway to a point in which it is not possible to gain any more signal even with an infinite exposure just for the fact that there are not enough photons that can be resoved 'cos the flux is mabye too low to be distinguished anyway from the minimum possible error of the system. I've just scratched the surface on that but I hope I've been almost clear on the concepts.

In the end, if our case is just to take a good picture and to generate emotions and curiosity just only rendering that subject in a personal or unusual way, there is no rules over Your satisfaction on the processing capabilities of Your datas! If this is the case then even a few seconds exposure could be enough!

Giovanni Paglioli

IOTD Staff

Jun 21, 2020 · Edited: Jun 21, 2020

I just want to add one more consideration on SNR Vs. f ratio, having a "faster" focal ratio does not improve tha SNR considering the same angle! This is just a Myth. Having a pixel that capture the same angle of the sky on a f3 system respect to an f8 system has no advantage on a linear acquisition device like modern CCD's or CMOS since the photon flux is a costant and no more photons can be added just for the f-ratio number. It is obvious that if you have a bigger angle than, the area is bigger and, 'cos photometry takes in account an area, a bigger number of photons could fall on a bigger area! Thant said, also the same ratio of unwanted noise deriving from the sky-noise will fall in to that area so that their relationship remains the same. Random photons on the other hand sums with the square root rule so can be mitigated by an increased number of exposures or with a longer one. The only way to increase the SNR on the same considered angle, reguardless if the system is f2 or f10, is to increase the collecting area! In other words if You want to increase the SNR on a onsidered angle, you have to have a bigger diameter telescope.