Old Data - to Keep or Not to Keep

I usually keep everything as long as I have space on my NAS.
Trying to reprocess old and bad data may still be interesting, in my opinion.

First, it's always nice to produce an image from a session.
More importantly, it's a good exercise for one's processing techniques.
And lastly, depending on the weather, this might be the only data left to process…

I just had a holiday for Christmas, and unfortunately, it was cloudy almost all the time.
So, I decided to go back to my old unprocessed data.
It was satisfying to salvage some of my failed sessions and produce an image.

The data for some of them was not that bad, but my lack of skills prevented me from stacking and processing the images correctly at the time.
The data on some others was awful, but with a little more experience, I could produce at least something.

The final images are not necessarily good in any way, but I did learn new things, which is always great.
In the end, a not-so-good image is better than no image, in my opinion.

Otherwise, it's just wasted time outside in the night.

The difference with keeping your grade school art projects is that you cannot improve them in any way.
You still can with old data.

Now, obviously, I started taking pictures in 2024, meaning the amount of data is really manageable.
I think it really depends on how far back your old data goes.

If the outcome I get from a data set is exactly what I wanted, I will SOMETIMES boot the raw frames, but keep the fully integrated/calibrated master file.
However, if I find that the result is not what I'd hoped for, I will zip up the lights, darks, flats and flat darks for the image and store them away with the NINA target information so that I can come back to the data next year, point to the exact same coordinates and add more data…

I have some data sets from 2009 still here on my hard NAS, but then I have images I made at the start of 2024, where I've just drawn the line in the sand, got rid of the raw data and I just have the unprocessed, calibrated/integrated master light in case I want another shot at processing the data.

I then keep Full res, 1/2 res and 1/4 res copies of all the processed images.

Old data doesn't necessarily mean bad data.  State of art camera 10 years ago can be almost as good as today's.  Light pollution 10 years ago was much better.  So old data can be as good as new ones.  If they are of good quality, keep them of course.  You may or may not need them in the future.  But if you do need them one day, you don't want to realize that you had deleted them a few years ago.

I've only been at this for 5 years so I don't really have any old data, but as you said storage is really cheap.  I keep my lights and calibration masters.  I also tend to keep my processing lights (calibrated, cosmetic correction, debayered) so I can add data over the years.  Generally use a 20TB drive for my data and another for my retrospect backups.  Seems to work well so far.  Cheers!

This is a very personal thing, I guess. Whatever you feel comfortable with I would say. 
For me, I keep my old raw data, my pixinsight project files and final output files. All stored on both a local SSD and a cloud service. Storage is cheap these days, so I see no benefit in throwing it away. What is very important in any kind of archiving is to keep structure. If it is impossible/very difficult to find things back, you will never look at it again. But if it's readily accessible, you might run into a use-case at some point.

For me it is a matter of piece of mind. If at any point there is a use-case for it, it would be nice to have the data. Or perhaps better stated, if there is ever a use-case that's interesting, I'd be quite bummed if I had thrown the data away for the sake of freeing up an SSD.

The most obvious use-cases are of course the re-processing with better tools, trying out alternative colour palettes, etc. Also you develop your own taste over time. The way you process your images now is probably different than 5 years from now. Not sure if you want to re-process all images based on a new taste, but perhaps there are one or two gems for which you're be happy to go through the trouble.

There's one type of use-case that always lingers a bit in the back of my mind as well. How about the things that actually changes in the night sky. Think of novae, asteroids, double stars, etc. Most of what we see changes at time-scales way longer than what we could realistically capture in a life-time. But not all. Novae of various kinds are obvious examples. When AT2019abn flared up in M33, I was lucky enough to have data from before the event, so I could make a 1:1 comparison. Comparing with the final image would probably also work, but in such a case it is nice to have the data from two periods processed the same way, giving maximal focus on the event itself.
Besides novae, it might be interesting to see how a Crab Nebula would look like in 10 years time, or a double star system with an orbital period of 20 years.    

Just my 2 cents.

I have digital astro data from a first production run STL11000m camera I purchased back in 2004, more than 20 years ago. This data was taken using wider field equipment from dark skys when i was traveling to do imaging. These data are  still good and I go back to the data, using new processing tools and looking for objects to image with my newer-larger equipment. I consider this a valuable resource. All of this is on-line residing on several  NAS drives, and backed up on blu-ray DVD's.

I still have Astro-image film data, from before the digital age, as far back as the early 60's.  Its value is limited, but provides an emotional connection to younger years.

All this takes very little space and costs very little to keep. Your choice, but for me I kep it all…

Gary Seven:
A 1 TB flash drive easily contains all of my old Lights, Master Flats, Master Darks, and Master Bias files. Toss it in the safe and forget about it.

According to the Beta 5 Computer instructions, all Class 1 Supervisors must store all gathered data indefinitely via the Beta 5 Exceiver Circuit; *however* Class 1 Supervisors "Can" override previous instructions on the Beta 5 Computer !

So I guess it is almost unanimous. Keep the old data.

As to mixing it in with new data, I can always give it a try.

Once I am happy with the integrated image from an observing run, I delete all intermediate data files (I mainly use PixInsight, so there is a considerable volume) and archive the original (acceptable) lights and the various intermediate and final integrated images.

My logic is that I can always re-run the processing sequence from the original light frames, or test some other new bright idea, which I do on occasion. I find this approach cuts down the archive data volume requirement by at least a factor of twenty - my original light frames are ~50Mb, while my intermediate file sets are all -300Mb per frame.

Of course, there is an implicit assumption that future software will be able to read and process archival formats, but that is probably a reasonable risk (says he who lived through Y2K in the IT industry).

Regards

Tim Anderson