How do you deal with multi-night project data? - Forums

Topic starter

Oct 30, 2021

Hello community,

during the last two weeks, I tried to image IC63 on purpose on a multi-night project. It may not be the easiest target. I've done multiple night shots before, but most of the time it happened more or less accidently. This time, I planned for the moonphase, what data to take and the way to combine the data to a final image. So basically it was fun beside the fact, that I am quite unpatient to wait to get the final result

.

However, I took a lot of data during 5 nights. I had about 30 hours of data in total. But I realized, that the conditions during all of the nights were quite different. Tha main problem are halos due to moisture in the air. They differ not only from night to night, but were inconsistend during the single nights also. In the end, processing the image was much harder than I thought of and I ended up taking only part of the data (about 14 hours) to get the final result. This was done to keep the halos of the bigger stars under control. This was a new dimension of problems I had to face.

So my question is, how you deal with the data you get from multiple nights. Did you experience such things, too? Is this a matter of the season and it will be less than a problem in another time of the year? What other problems one may have to take multiple-night shots?

This question is ment as an exchange of experience that could help not only me, but other photographers, too.

Thank you, for your answers…

Clear skies

Christian

Helpful Insightful Respectful Engaging Supportive

Sean van Drogen

Oct 30, 2021

Hi Christian,

For multiple nights the main tool now in my arsenal to get the best master to process is the NSG script in PI.
Still process and align everything manually to begin with. It does mean quite a bit of data will get rejected, but usually no more than 10%. Think this might be higher if the night conditions vary a lot. Once did a test combining data from not just different nights but different years. With this the rejection rate was double 20% so still quite acceptable to get a nice clean master. From your write up it seems the conditions were at extremes to have to reject that much data.
Think for bigger projects I think we have to accept that the more data will be collected more will be rejected to get the best result.

Just my 2 cents

Clear skies,
Sean

Helpful

David Nozadze

Oct 30, 2021

Hi Christian,

My only multi-night (i.e. more than two) project was M31 this summer. I took subs for 7 nights, but each night I gathered data in all channels (LRGB Ha and OIII).
Naturally, conditions varied quite significiantly on each night.

But, I decided to disregard the separation by sessions. I just grouped subs for each individual channel and then selected the very best out of those with PI's Subframe Selector. The "abundance" of data allowed me to be "merciless" in the selection. Thus I ended up with rounghly 7 hours wors of data, out of almost 30 hours total.

Here's the result. It is not perfect, ofcourse, but I think it is not too bad either.

https://www.astrobin.com/6irxc1/D/?nc=collection&nce=4766

Clear skies and looking forward to seeing your IC63. Mine did not come out too well

Still a difficult challenge for a novice like me.

David

Helpful Engaging Supportive

Christian Großmann

Topic starter

Oct 30, 2021

David Nozadze:
Hi Christian,

My only multi-night (i.e. more than two) project was M31 this summer. I took subs for 7 nights, but each night I gathered data in all channels (LRGB Ha and OIII).
Naturally, conditions varied quite significiantly on each night.

But, I decided to disregard the separation by sessions. I just grouped subs for each individual channel and then selected the very best out of those with PI's Subframe Selector. The "abundance" of data allowed me to be "merciless" in the selection. Thus I ended up with rounghly 7 hours wors of data, out of almost 30 hours total.

Here's the result. It is not perfect, ofcourse, but I think it is not too bad either.

https://www.astrobin.com/6irxc1/D/?nc=collection&nce=4766

Clear skies and looking forward to seeing your IC63. Mine did not come out too well Still a difficult challenge for a novice like me.

David

Hi @David Nozadze,

thank you for your answer. I think your M31-image looks great and I think 7 hours trash out of 7 nights is quite a good ratio. My image of IC63 can be seen here.

https://www.astrobin.com/83311q/

The halos around the big stars come from different subs. To get a result without them, I must have rejected about 80 percent of the data. I will try this target next year again. But now I want to concentrate on some more things in the sky before the subjects disappear for the next months.

CS

Christian

Respectful Supportive

Joel85

Oct 30, 2021 · Edited: Oct 30, 2021

"So my question is, how you deal with the data you get from multiple nights. Did you experience such things, too?"

Hello, (sorry for english , hopping it will be understandable)

I have a short experience on several nights capture (M31 11H, M33 27H, M101 18H ) / same weeks of the years (following days Day + 1 +2 etc...). I have never used frames from year -1 yet / Newton Carbon UNC 200/800 / ASI071 color

- I do not touch the setup between different nights (remain installed and covered )
- Astrometry : precise photo framing (always the same "center" for one object) / I identify a star in "the middle I want" and run astrometry on this star (more precise but it works well often with the object reference NGC or other)
- Flats when it is ended (after several days ) : could be made every moring but not necessary (depending on local contexte)
- The camera + corrector are screwed on crayford --> collimation is easyer + always same position ...
- A fixed concrete base column for the mount (HEQ5) / no tripod (it can work with a tripod but it can move too... )

During the night, moisture is sometimes > 90% (&more) where i am living (autumn/winter ...swamp...) : a dew hearter is used on ... primary mirror

: it works very well

+flexible dew shield. Without this , it would often be impossible , even for one night sometime ... (i have checked and made tests during summer : the heater does not seems to blurry image / very small & thin difference but ok !!! ) . No moisture any more on primary & secondary mirror : better lights from one day to another (only seeing & sky differences)

Processing & stack

- Pixinsight
- FWMH : a selection is made (manual) after an ascending sort on FWMH of all lights --> the best one is used (= a reference align)
- 100% succesfull (align) / Ransac parameter = 2

I am a bigginner for "big stack on several days" , there is probably better advise than mine, that's all I can share with you , hoping it can help !

https://www.astrobin.com/nmxvcr/C/
https://www.astrobin.com/grosq4/C/
https://www.astrobin.com/x1ifqx/

Clear Skies !

Gustav Lundby

Oct 31, 2021 · Edited: Oct 31, 2021

The simple way is to rely on Astro Pixel Processor to take care of it. Just enter the data as different sessions, each session with their calibration subs, possibly assigning data to different channels (Ha, Oiii, R, G, B, L). My experience is that even with different FOV and diferent optics even different sensors and perhaps different pointing center, APP registers and integrates it all. You may choose to get it all (like kind of a mosaic) or pick out an area of interest and just get that. You may also get results from each session separately during the process (for merging later in your editor).

The more advanced way is to integrate each session separately, inspecting and rejecting bad stuff. Then using advanced editing using layers and filters to achieve the wanted result.

Example integrating Sony a5000 on TS65 f6.5 quad with Optolong L-eNhance with Sony a7 on vintage Takumar 4oomm f5.6 :

https://astrob.in/mescvk/C/

Helpful

Christian Großmann

Topic starter

Oct 31, 2021

Hi again

and thank you for your help. But I may not be precise enough. The problem does not lie in image processing itself. I didn't change the setup, used the same camera and with plate resolving, it is easy to find the same angle of the sky again even with the rotation set precisely as the night(s) before. I've done it before and it worked. I also find, that PI does also a good job stacking the images. I also used the Normalize Scale Gradient script for each seperate channel (but tried it for the first time).

Last night, I took some images of M33 while I realized, that the bright stars may be the issue. I had some fog and some clouds again and the halos around the brighter stars in this region of the sky are much more consistent this time. But the stars around M33 are clearly not as bright as the ones next to IC63. So I think, that the halos surely vary the same relative amount as the halos from the brighter stars , but the problems are by far not that obvious. If you got a halo that takes 5% of the final image, its variations are much more visible than the fainter ones.

Maybe that is, why targets with really bright stars are hard to photograph

Linwood Ferguson

Oct 31, 2021

The only thing I can add I think is "Cull".

I cull ruthlessly, if it means taking a few more nights, that is what it takes.

Frequently I will go into subframe selector (or blink, but usually sfs) and see this hump – usually a bad hump, like in fwhm, or a bad dip like star count. Often I will find a whole night, due to seeing (or whatever) is just a lot worse than the other nights. Often it is not a full night, might be just after moonrise, or below a certain altitude.

If it's a LITTLE worse, I just let it average in with lower weight, but I am finding my images are better if I am a bit more ruthless. The GE/Jack Welch philosophy of imaging.

Helpful

Lynn K

Nov 1, 2021 · Edited: Nov 1, 2021

Hi Christian. If I understand your question, you are concerned with how to edit/select best data from multi sessions. I think that will vary a great deal based on location, equipment and personal bias towards what is acceptable data.

I always do multi sessions and have been for years. Not out of choice, but necessity. I live in Bottle 7 skies have grown to want the best image I am capable of. I do primarily narrow band with a permanent setup. But even before I had a permanent setup, I would still do multi sessions.

The first step, whether permanent or portable is to be sure your camera rotation and framing is the same. If the camera has been removed, it is more difficult to achieve similar rotation. So, if possible leave the camera imaging train attached to the scope. Of course, with a permanent setup, this will not be an issue, UNLESS you image another object and change rotation. In that case, one may fine a remote rotator necessary. I do. If you image setting next to the setup, your hand is the best rotator.

The following is my procedure. One slewed to the object take a short image binned to max. If doing nebula, filter in Ha. Open an image from the last session and compare composition/rotation. Put crosshairs on both to compare center. If your program has a centering option, that will help. I just choose a bright star or two and compare location of the stars in the two images in respect to the crosshair. I run 4sec continuous images binned 4x4 in Ha, mind move the scope via ASCOM hand controller still the stars or in similar position in both images. Then I am composed and ready to start the session.

BUT, more to your question. After each session I will go through the previous night's subs. For me that more likely be using one filter (Ha, OIII, SII). I will calibrate and stack. I may not have flats yet, but this process is to determine S/N, not field curvature. I do CCD imaging, and my subs are usually 10 to 15 min. On a single night, I will end up with 10 to 20 subs. Unlike a CMOS imager with 60 to 100. So, going through each sub is not overwhelming. I use MaximDL and do a DDP process on the stack. That is a non-liner agulrythem that stretches the image so I can see the object. I may push it a bit and see how bad the noise is. Other post processing programs probably have something similar under a different name. I then determine how many more subs I will need to lower the noise to acceptable levels. I may even take the stack in to Photoshop to see how bad the noise is, and how difficult to soothe out. Probably more subs will be necessary, and I will continue with that filter until happy with the noise. This process will continue through the OIII and SII. Then I will get some short RGB on the stars. I will do all Red, Blue and Green in one quick session, they are usually 1 to 2 min and little to no issue with star roundness.

I should also mention that I may try different stacks of the data, some being more selective and see how that changes the results of S/N. It's a tradeoff of more subs rendering lower noise vs selected subs rendering better resolution.

As far as your star halos, I am confused. Usually halos are caused by internal reflection off the filter and chip cover window. And the reflection back and forth causes the halo. Consequently why, usually only occurring with bright stars and off center towards the outer frame.
My only personal solution to that, is better filters using better non-reflective coatings.
A lower F ratio will increase the halos, but I am confused why atmosphere conditions would be a factor. I'm inclined to think it is a infocus/out of focus issue. Your scope may be creating an excessive diffraction ring from the stars, which will worsen when out of focus. Scopes changing focus over the session is a typical problem. Better corrected optics will focus more of the light into the starry disc . Poorer corrected optics will not bring all light to focus in the starry disc and more will end up in the diffraction rings. That will increase with poor focus. But that is only a guess. Very poorly focused star will cause a diffraction ring donut. But with that there is no star point at all , just a small donut for each star.

I hope this addresses your issue in some way,
Lynn K.

Christian Großmann

Topic starter

Nov 1, 2021

Lynn K:
Hi Christian. If I understand your question, you are concerned with hour edit/select best data from multi sessions. I think that will vary a great deal based on location,equipment and personal bias towards what is acceptable data.

I always do multi sessions and have been for years. Not out of choice, but necessity. I live in Bottle 7 skies have grown to want the best image I am capable of. I do primarily narrow band with a permanent setup. But even before I had a permanent setup, I would still do multi sessions.

The first step, whether permanent or portable is to be sure your camera rotation and framing is the same. If the camera has been removed, it is more difficult to achieve similar rotation. So, if possible leave the camera imaging train attached to the scope. Of course, with a permanent setup, this will not be an issue, UNLESS you image another object and change rotation. In that case, one may fine a remote rotator necessary. I do. If you imagge setting next to the setup, your hand is the best rotator.

The following is my procedure. One slewed to the object take a short image binned to max. If doing nebula, filter in Ha. Open an image from the last session and compare composition/rotation. Put crosshairs on both to compare center. If your program has a centering option, that will help. I just choose a bright star or two and compare location of the stars in the two images in respect to the crosshair. I run 4sec continious images benned 4x4 in Ha, mind move the scope via ASCOM hand controller still the stars or in similar position in both images. Then I am composed and ready to start the session.

BUT, more to your question. Afer each session I will go through the previous night's subs. For me that more likely be using one filter (Ha, OIII, SII). I will calibrate and stack. I may not have flats yet, but this process is to determine S/N, not field curvature. I do CCD imaging, and my subs are usually 10 to 15 min. On a single night, I will end up with 10 to 20 subs. Unlike a CMOS imager with 60 to 100. So, going through each sub is not overwhelming. I use MaximDL and do a DDP process on the stack. That is a non_liner agulrythem that stretches the image so I can see the object. I may push it a bit and see how bad the noise is. Other post processing programs probably have something similar under a different name. I then determine how meany more subs I will need to lower the noise to acceptable levels. I ma
y even take the stack in to Photoshop to see how bad the noise is, and how difficult to soooth out. Probably more subs will be neccassary, and I will continue with that filter un till happy with the noise. This procees will continue through the OIII and SII. Then I will get sonne short RGB on the stars. I will do all Red, Blue and Green in one quick session, they are usually 1 to 2 min and little to no issue with star roundness.

I should also mention that I may try different stacks of the data, some being more selective and see how that changes the results of S/N. It's a tradeoff of more subs rendering lower noise vs selected subs korendering better resolution.

As far as your star halos, I am confused. Usually halos are caused by internal reflection off the filter and chip cover window. And the reflection back and forth causes the halo. Consiquenyly why, usually only with bright stars and off center towards the outer frame.
My only personal solution to that, is better filters using better non-reflective coatings.
A lower F ratio will increase the halos, but I am confused why atmosphere conditions would be a factor. I'm inclined to think it is a infocus/out of focus issue. Your scope may be creating excessive refraction ring from the stars, which will worsin when out of focus. Scopes changing focus over the session is a typical problem. Better corrected optics will focus more of the light into the starry disc . Poorer corected optics will not bring all light to focus in the starry disc and more will end up in the refraction rings. That will increase with poor focus. But that is only a guess.

I hope this addresses your issue in some way,
Lynn K.

Hello Lynn,

thank you for your answer!! It's great to hear, how you are working. Here are some more thoughts about it...

I do not have exactly a permanent setup. I have a place in my garden, where I put the mount always on the same position. The feet positions of the mount are marked by crosses on the ground. I use three stones with scores on the floor for that. While the mount is always on the same place, I change scopes and cameras on purpose. I printed some markers for the rotation angle of the camera and its accessories (+/-90 degrees) and sticked it to the coma corrector. A mark on the focuser tube, where the coma corrector slides in, marks a field rotation of zero degree. This way, I can set the exact rotation angle before it even gets dark outside. I did this on all of my scopes. Although it was a bit difficult to measure and create the angle scales, it works really great. To check the angle, I use plate solving and I really don't care, if the field of view is slightly off, or should I? I think, with plate solving and goto, the images must not be aligned super correctly. Dithering will change the position slightly anyway. The rest is done by the stacking software.

The halos I mean are not coming from the filters. If there is moisture in the air, it will stray the light a bit. This is the same as if clouds are moving through the field of view (of course it is, they are moisture or fog). The light gets more and more diffuse as it passes through those layers. That produces some weird halo-like glows around the stars. I am sure that it isn't caused by dew on the mirrors or the sensor. I checked it several times and all elements (even the secondary mirror) have dew heaters. I never had issues with dew.

So maybe the weather conditions were worse than ever before.

CS

Christian

Helpful Engaging Supportive

Lynn K

Nov 2, 2021

Christian, it seems you have things worked out. I have done similar processes. But, there is nothing we can do about the weather except move. I don't experience that kind of fog. If it is foggy here, it will be fairly thick an stay awhile. My issue is typical clouds coming through. They will block out the light, mess up guiding. The the result will be a whitish washed out sub if the cloud is fairly thick or just real noisy if it is faint. But I have never experience stars like yours. But I can see how a foggy atmosphere would cause halos around bright stars just like it does the moon.

I bough a used cloud sensor years ago, but never set it up. It has to be calibrated to how sensitive I want it to be. I guess, one could set it to activate the alarm even with fog. I bought a ZWO ASI178 camera to take video images of the sky. Then I can monitor the sky transparency from in the house. But I haven't set that up either. I have Cad5 cables running into the house, so I can run the setup from inside. When I go to a dark site, I take a trailer and do the same.

Sorry for all the typos in the first post. I was doing that on an android tablet and the system keeps auto replacing my typing, which is poor anyway.

Lynn K.

Supportive

Robert Gillette

Nov 5, 2021

Christian,

It sounds as if your main issue is not imaging on multiple nights but your halos. I've combined images over a span of several years. The only real challenge is aligning new images with previous ones within 1-2 arc minutes, and getting camera rotation approximately right, especially when imaging on a different side of the meridian.

I use the same image from the original set as a reference for alignment. If conditions are poor, I just don't do it. But even in the conditions here in northern New England, where seeing is rarely excellent, I've never experienced halos.

CS, Bob

Well Written Helpful Concise