Integrating world wide data

I think something like this has been tried. Try to sort the Big Wall by integration time, and maybe there's something there!

The Sort menu, top-left on the orange menu bar!

Hi Björn,

there were already several attempts on that in the past. To sum it up: It's possible but data integration is a challenge :-)

Have (for example) a look here:

https://astrob.in/z2vojq/E/

Cheers
Frank

Hi Björn

I tried doing that some years ago. I had the same idea you are talking about. What if we combined data from all the astrophotographers into one?

The thing is, as you already know, that an image is just a set of data. Each pixel contains information (signal) about the object being imaged, and on top of that the pixel also contains noise (all sorts of noise). Therefore you can calculate a signal to noise ratio (SNR). This is the basic principle of astrophotography, because we basically image an extremely faint object like a galaxy on an even darker background. This means that once you gain the signal, you also gain the noise.

The longer you expose, the better the SNR. However it doesn't scale linear. Typically if you double the exposure time, you will get an increase of SNR equaling the square root of two. An increase of exposure of 4 times, gives an increase of 2, and so forth. 100 times longer exposure "only" result in 10 times better SNR, so there is some sort of diminishing return on the time invested.

The data that any telescope/camera combination output is dependent on the setup, and also on the operator (photographer). Some are better than other, but when I started having this idea, there was one thing that kept me going. As long as you can see the object of interest in the image (the faint fuzzy), then the image contains valuable information that can be added to similar information from other images. That was my central philosophy.

I started out arranging a collaboration of roughly 10 danish amateur astronomers, through the forum for the Danish Astronomical Society. We collected data from the relatively large (less problems with resolution) planetary nebula called Jones Emberson 1. The result can be seen here:

https://www.astrobin.com/136820/?nc=user

This really motivated me, but gathering the data and keeping track of them were, let's call it, a challenge.

Next, I went all in, and collected as many images of the Andromeda Galaxy as I could. Several thousand jpg images from all over the internet. I started my astrophotography using homemade scripts and code, but quickly turned towards Pixinsight. That is the best software for combining large amounts of data hands down (sorry Maxim et al). Using Pixinsight I started experimenting and it literally took months and months. Stubbornness pays in the long run. 

First thing to do is to chose one master image with a good wide framing. Then you align all the other images to the master. Use distortion correction! Also the master should contain as little distortion as possible. You'll learn that along the way.

Once the images are aligned (registered), you stack (integrate) them. A basic average stacking is a good start, and will get you a long way. Then you can experiment with all sorts of pixel rejections. For normalization I typically use scale and offset, but you can do without.

The final stack will blow your mind. Once you start stretching the stack you'll realize how high the SNR can get. This is a later version of M31 that I made, but it does show the potential (300+ images stacked):

https://www.astrobin.com/120204/?nc=user

The method makes it possible to create very deep images of the night sky. I've stopped publishing my images, but I still do them once in a while, when I want to see something, and I can't find a proper image. Recently I've been looking for galaxy clusters, and I've "seen" pretty far away using this method.

The method was dubbed "Crowd Imaging" (CI), by one of the members of the Jones Emberson team. This name also gives a hint at one of the very time consuming parts of it, being keeping track of all the participants. I decided to avoid copyright issues by only using Creative Commons licensed images, and I knew how much work each and every amateur astronomer put into making each separate image, so I decided to keep a spreadsheet with the names etc. of everyone involved. With several hundred people, that's time consuming. You might argue that this is covered by "Fair Use", but for the experiments I did and published, I think it's fair to give credit to everyone.

I highly recommend trying it. The method definitely works, and it has both strengths and limitations. You can get very high SNR, and if you start assigning different weights to each image based on SNR, FWHM etc. you can make different stacks that you can combine. An example could be a high SNR stack for that dark areas of the field, and a sharper one for the galaxy core etc.

Good luck, and feel free to ask. I'm pretty busy with other work right now, but I'll happily try to help. Most of all, this is all about diving into huge amounts of information (literally physical information in bits), and trying to optimize the method. That is the challenge, and that is where the real fun is found

I love the concept and really like the thought of contributing to a larger project in a meaningful way...

An interesting alternative, I think, would be to have a small group image targeted areas of the sky and stitch their results into a mosaic. It would be interesting, for example, if the RedCat+ASI183 owners grouped together for a deep integration wide field of, say, Cygnus, or Cepheus, or any other constellation beginning with "C". :-)  Of course we may need a supercomputer...

Any Cat+183 owners interested?

@Björn

I used to do Crowd Imaging on an old 2008 Mac Pro. That definitely brought the machine to it's limits. Nowadays I have a R3950x, SSD raid and 64 GB mem. That speed it up a lot, but shift stacking 1000+ images at 4K res still takes time. I think I can do it in an evening, but stacking more than 200 images is normally a waste and that is doable in a couple of hours.

Back in the day, I communicated with some Canadian astronomers, that had been trying to use a web crawler to gather image data of the sky, and then combine it into one giant spherical map. They got stuck with the shift stacking and pixel rejection if I remember correctly, but I had that one solved, at least as a proof of concept in Pixinsight. Later we lost contact, and I haven't heard about them since then.

The thing I mostly use Crowd Imaging for today is creating very deep fields. Even though jpg compression eat a lot of information, there is still a lot of information to be found down there, once you stack a lot of images. Some of the galaxies away from the Milky Way plane have a nice clear field, and behind the galaxies you can find all sorts of interesting stuff. You have no problem seeing large galaxy clusters several billion light years away in detail.

One of the limitations of Crowd Imaging is that the popular night sky fuzzies are welly documented, but you find some parts of the sky where there is no amateur images available. Another is the limited resolution compared to large modern telescopes with wobbly mirrors. Also, since amateur astro photography is becoming "older" by the year, some of the fast moving stars get rejected just like a satellite. I have been looking for a way to sort a set of jpg images with regards to time, but haven't found it yet. That would open up for animated deep sky images, which could be interesting. A picture says a 1000 words.

If you use Pixinsight, a good start setting for ImageIntegration is:

Combination: Average
Normalization: Additive with scaling
Weights: Don't care (all weights = 1)
Scale estimator: MAD
Evaluate noise: Off

Rejection algorithm: Sigma Clipping
Clip high
Sigma=5

Clip low range: Enabled (!) This one is important.
Range low: 0

Be aware that Pixinsight create an alpha channel when StarAligning a BMP image, so to integrate those, first delete the alpha channel. Also convert mono images to RGB. Both are easy to recognize in a file list, due to file sizes.

StarAlign is pretty solid, but some images are simply too far off the average and the framing, so typically you will lose 5-30% of the sub images.

Supersample the master to 150-200% of your wanted final resolution and downscale the final result. That will demand more computer power, but it is worth it with regards to the final result.

Cs

Morten

If you start something like this, I am in also. The only issue is that I cannot guarantee that in my area the weather will cooperate within a week. If it can be extended I am in.

Hi everyone! 

It seems that quite some people are interested in that. Of course it's one thing of having an idea and realizing it is another story.

So to come some practicalities: I thought it might be a good idea to create a group on AstroBin where people can organize these crowd imaging sessions/projects for specific targets and equipment ranges? People could spontaneously arrange and agree on collecting their data in whichever manner they want.

What do I mean by "equipment ranges"? I thing to create reasonable results from raw data (I guess it would suffice to share the calibrated raw files), the imaging parameters should match within certain ranges. I haven't tried it but I would assume that combining luminance data with 0,5"/pixel with 3"/pixel isn't very helpful. Also the fields of view shouldn't be extremely different, as the total integration will be limited to the smallest FOV (including the image rotation: if somebody shoots with North in horizontal direction (landscape) and somebody else in vertical (portrait), the intersection will lead to a smaller FOV). So I could go on with these things but I think it's something people need to agree on when setting up. 

Björn