What files do you save, long-term? - Forums

Topic starter

Mar 21, 2025

Currently, I'm saving my master files, the PixInsight project files, the result as a high-quality TIFF (maybe 250MB), and finally the resampled image as a PNG for uploading (maybe 25MB). I've started to keep the masters so I can re-process my images as my skills improve.

What files do you save, long-term?

Well Written Engaging

Alex Nicholas

Mar 21, 2025

For each dataset I image, I keep the following files.

1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)

If HDD space was tight, I'd get rid of the master lights first… You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.

If HDD space was still tight, I'd buy more HDD space.

Helpful

Noah Tingey

Mar 21, 2025

Alex Nicholas:
For each dataset I image, I keep the following files.

1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)

If HDD space was tight, I'd get rid of the master lights first... You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.

If HDD space was still tight, I'd buy more HDD space.

What's the reason to keep the uncalibrated lights + the calibration masters, versus keeping just calibrated lights?

Well Written Engaging

Dunk

Mar 21, 2025

I personally keep all my lights/flats/darks.. Tools like WBPP are constantly innovating and improving and I want to be able to go back to the source if needed.

HDD's are very cheap.

Jesper

Mar 21, 2025

I have a 4TB HDD and i store everything on it. Mostly i do planetary and have files of around 20GB.

Once full i start deleting.

daywalker

Mar 21, 2025

If it gets to the stage that i process it I keep it,(unless I think the data is hopeless). Clouds don't allow much data generation anyway so If i doubt its kept,as theres always the chance that processing software and my processing techniques will improve later.

Bill McLaughlin

Mar 21, 2025

AmyWarble:
Currently, I'm saving my master files, the PixInsight project files, the result as a high-quality TIFF (maybe 250MB), and finally the resampled image as a PNG for uploading (maybe 25MB). I've started to keep the masters so I can re-process my images as my skills improve.

What files do you save, long-term?

I save all files including the PI project files for maybe a week after I finish just in case I discover that something needs to be changed. After that I save the final integrated components, the calibration masters and the calibrated files. I delete everything else. This is on my processing PC. The original lights are still present on my acquisition PC although in 30 years I don't think I have reprocessed more than 10 or so images out of hundreds. Too many objects, too little time.

I do move the files I save on the processing PC from a 4 Tb fast M.2 to a 20 Tb mechanical drive when the week is up so as to free space on the fast working drive for images yet to be processed.

Helpful

Kyle Goodwin

Mar 21, 2025

I keep everything I capture raw, plus masters and my processed results. I don't keep calibrated/registered/whatever preprocessed frames once I'm done with a target because if I reprocess later I will regenerate those.

Tony Gondola

Frequent contributor

Mar 21, 2025

All the original light and calibration frames plus the final render for the subject. Everything else goes in the bin.

Well Written Concise

Rich DeMidio

Mar 21, 2025

I keep my lights, flats, bias, and Pixinsight Projects (one project per target). I have not been using darks since I have a cooled camera and shoot at -15 or -20. If I did have darks, I would keep them as well. One thing I did not see mentioned was backup. I have iDrive and it will automatically backup any file dumped into a folder structure (and subfolders). I don't backup the PI project so that is a risk.

I also have a folder structure in which all of my final jpgs for publishing are stored and backed up. Finally, I have a folder structure for all of my master lights which are also backed up. So, my process.

Blink all of my subs removing bad ones.
Move flat masters to folder structure (backed up)
Move good subs to a permanent folder structure (gets backed up)
WBPP to produce the master
Save the master light to the permanent folder (just the integration, I do not keep the high/low rejection images).

Regarding the WBPP output folders, I save them until I need to cleanup my disk space. I place them on a 2TB internal drive. They are not backed up.

Hope this helps

Rich

Helpful

Médéric Hébert

Mar 21, 2025

I keep every lights, calibration frames, final tifs and Gimp project. Every thing else gets deleted

Observatório Astrográfico do Boqueirão

Mar 22, 2025

Hello Amy,

Currently I keep all my lights and calibration files. In the beginning I though just keeping the masters was enough to save space, but I've learned the hard way not to do it anymore, mainly because stacking to create them tends to change from time to time, and what's its good now, it's not tomorrow. So the best policy is to keep the originals. From that you could do anything you want form them.
This was also something that I've learned from my boss related to our company archive. Keep the files as much original as you can. Because after that, the process is "destroying" and "degradate" the RAW product, that could be photo, video, audio, analog or digital.

Another suggestion: don't save data in larger disks. Do the exact opposite. Minus disk space, and more disks. Yep it's expensive, but if you value your archive, this way you minimize the data loss if one of them stopped working, loose data, etc. It's more easy to loose a smaller full disk with data, rather than one with a lot of them. Another value lesson that I've learned with the years.

Best regards,

Cesar

Helpful

Ian McIntyre

Mar 22, 2025

Saved everything the first 3 years doing this. Now I delete the WBPP calibration, register, and debayer folder. Save sorted/blinked original raw FITS, masters and processed finals with stars and starless, along with any intermittent experimentations in native PI .xisf. And obviously the portable finals for publishing, which are fairly inconsequential for capacity usage when compared to the raws and native finals.

Yeah, I was starting to really hit the ceiling and tired of spending for more storage. Dumping those 3 batch processing folders saves a ton. Just costs time if you want to restack later. Which, I have actually done more than once. But what the heck. Just set it up to run while sleeping or at work.

Helpful

Richard

Mar 22, 2025

During import, I blink my raw subs and delete any bad ones (eg cloudy cover, star trailing). I keep all raw subs and flats.

I make a bias/dark library and once I've created masters, the raw files are placed into a ZIP folder to reduce size.

After WBPP, I keep the master lights but discard the lights in between (eg calibrated, cosmetic corrected, registered) as these can always be generated again in future from the raw subs. Now I have a master flat, the raw flat frames are then put in a ZIP folder to reduce size.

I will keep a Pixinsight project as it saves your history of processing, so if you ever want to tweak your final image using a newly learned technique, you won't have to start from scratch. When a project in complete, I export the final image as a JPEG for uploading to websites.

I may revisit the above approach and keep the calibrated light frames and ZIP the raw lights because I may have a project spanning a couple of years which uses different master darks which can be annoying to group with keywords during WBPP.

Helpful

Alex Nicholas

Mar 23, 2025

Noah Tingey:
Alex Nicholas:
For each dataset I image, I keep the following files.

1. All my lights that pass selection processes.
2. Calibration masters used (per image)
3. Master Lights (and any drizzle integrated master light).
4. PixInsight Project containing final Starless and final Stars image, plus the individual channel data, and the rescreened result.
5. Final processed image as *.tif (16bit unsigned integer)
6. Final processed image as 100% scale, 100% quality *.jpg
7. Final processed image as 25% scale, 100% quality *.jpg (for posting online)

If HDD space was tight, I'd get rid of the master lights first... You can re-create those using the master calibration frames and the lights if need be.
Following on from that, I would next delete the PI Project file for any image that I considered "Complete" that I was 100% happy with.

If HDD space was still tight, I'd buy more HDD space.

What's the reason to keep the uncalibrated lights + the calibration masters, versus keeping just calibrated lights?

The reason to keep the raw lights and raw calibration frames is that stacking algorithms change over time. Should I want to go back and re-stack the the data later, utilizing new calibration techniques or stacking algorithms, I want to be able to do so...

part 2 - calibration masters is not 'calibrated light frames' its the master dark, master bias and master flat for each filter. I keep these per image, as if I want to extract an image's files and re-stack that data - I want to have the cals that I used for that specific image available, I don't want to also have to try to rememeber which date the flats I used for that image were taken on..

Once I'm done processing an image, everything other than the final JPG images are compressed fairly dramatically for archiving.

Helpful

Rigo1

Mar 23, 2025

For me files from 5 years ago I only keep the masters, have 15TB of data and still deleting files, cameras and telescopes and software is getting better and better and old ones are sold, so in my opinion old data will never be re-used (except to good ones) .

AstroRBA

Mar 23, 2025

Storage is cheap BUT it can still be a pain collecting it all and having a whole basket full of portable drives! Having said that, I keep all good subs and Pixinsight Masters although the chance that I'll revisit those files is slim to none …

danieldh206

Mar 23, 2025

After I have calibrated, stacked, and processed an image I zip the master calibration files with the raw fit files together. It does not take long to calibrate the raw files. This way I am storing zipped 16bit files which are half the size of the calibrated 32bit files. I only keep these zipped files for about 2 years and then I delete.
How long to keep raw data really depends where you image from. If you image from a place where it will take 3 years to get 20 good hours on a target then keep the data 4 or 5 years. If you are lucky and image from a place where it only takes a couple of weeks to get 20 hours of good data then you do not need to keep the data for very long.
Once you have a really good stacked master image it becomes increasingly more difficult to create a better stacked master. More masterLight processing experience and new masterLight processing tools make the biggest difference in the quality of the final image.

Helpful

Robert Sahari

Mar 23, 2025

I'm keeping all my lights, darks, flats and biases. Storage space (external SSD) is so cheap now that there's no reason to throw anything away.

I throw away all the process files however, after I've saved the final image. If I'm not happy or think I can do better at a later date, then I simply start with the original unprocessed data. I'd need to do this anyways, if I added more lights to a prior target.

Well Written Helpful Concise

Topcode

Apr 14, 2025

I keep effectively, all files which are generated any any point in my process (except of course, the temp files which are automatically deleted). I could probably cut down on data usage a lot, particularly with the files generated through PIPP as intermediates, but I am not going to, because I have many terabytes of storage lying around.

Paolo

Apr 14, 2025 · Edited: Apr 14, 2025

For every target I store:
* all light frames
* master dark
* master bias
* master flat
* PI project (so I have history, etc)