Data Retention (calibrated lights vs raw + library of calibration frames)?

Topic starter

Apr 24, 2023 · Edited: Apr 24, 2023

I'm curious what everyone else out there does in terms of archival and data. For the past year or so I've been keeping everything (lights, cal frames, masters, etc) As expected, this is getting much larger, so I've recently started dumping raw calibration frames and only retaining the masters. This has helped, but I'd love to hear what everyone else does

Do you retain everything? How?

Do you retain all raw light frames? Or do you save only calibrated light frames?

I'm thinking of only retaining calibrated light frames. Is there a well understood bit rate to do this at? 16, 32, 64? The idea of losing data haunts me.

I appreciate hearing any of your insights on this

Engaging

Ruediger

Apr 24, 2023

Hi Nick,

I keep all the raw data, plus the calibrated master. All intermediate data, which can be re-generated (calibrated, registered) gets purged after closure of the project. Not because of the data amount, but the time the backup needs to be replicated off-site.

CS
Rüdiger

Well Written Concise

Paolo

Apr 24, 2023

I keep:

raw light frames
calibrated masters (bias, dark, flat)

Nick Grundy

Topic starter

Apr 24, 2023

what's the average data size we are all holding? I'm sitting on about 2TB. Wondering how you all manage that? replicate offsite? NAS?

Engaging

Ruediger

Apr 24, 2023 · Edited: Apr 24, 2023

Nick Grundy:
what's the average data size we are all holding? I'm sitting on about 2TB. Wondering how you all manage that? replicate offsite? NAS?

Hi Nick,
my AP data is approx. 1.8 TB, which is actually not a big deal compared to the rest. I am keeping the data replicated two times locally (on- and offline) on real NAS Servers RAID6/ZFS Z2 and replicate one set off premise to another NAS via permanent IPSec tunnel.

CS
Rüdiger

Paolo

Apr 24, 2023

Thanks to the weather, I don't have to spend much for HDDs

I currently have a RAID 1 of 2 4tb HDDs, and I have plenty of space available.
I also backup a copy online (B2 now, something else past the 2tb).

Doug Summers

Apr 26, 2023

Keeping raw light frames (FITS) and calibration masters (dark, flat, flatdark), helps minimize storage but still allows for reprocessing (if desired). Otherwise, retaining xisf files begins to really suck up disk, especially as the number of images kept in your archive increases.

Helpful Concise

Brian Puhl

IOTD Staff

Apr 26, 2023

I'm glad someone posted this poll…. I'm starting to eat through hard drives rather fast… lol I've been debating chucking the raw files and keeping only the Pixinsight directory.

Nick Grundy

Topic starter

May 1, 2023

I've been thinking about only keeping calibrated light frames. (so I can be more in the habit of taking flats nightly, then dumping them regularly)

I've been trying to decide if I can save my calibrated light frames 16-bit integer FITS without losing any data. I've tested stacking from a calibrated set of 32 bit float FITS vs a stack of 16-big integer FITS, and I can't seem to tell the difference. Maybe one of you all knows the more technical explanation or answer?

I dropped a couple subs + the stacked HA images to see if you all can tell the difference.

I most definitely want to keep light frames, but i think restacking them on later dates is so much easier when I don't have to remember which flats/dark flats and such go with each date. (I tend to collect more data and restack from some older data or optics to see what comes of it)

I'd love to hear how you all handle this

https://drive.google.com/drive/u/0/folders/1TOpSVKz-kXbfmy1wKNXK4wzPpQ8Tx0j1

Helpful Insightful Respectful Engaging

Wei-Hao Wang

Perfect post

May 2, 2023

Raw files are the direct output of your hard works at night. Never delete them. Hard drives are dirt cheap compared to your other astro equipments. Buy several large hard drives to store the raw files and make backups. (Get a fast SSD for processing. Don't use hard drives for this.)

I keep all the raw files and master calibration frames (since they can be reused again and again). I keep stacked linear images. I also keep all intermediate processing steps toward the final, so I can go back to whatever step any time if I am not satisfied with the final. I don't keep calibrated single subs because they can be generated rather easily from raw and master calibration files when necessary (truth is this is rarely necessary), and also because calibrated subs take more space than a single stacked final.

Helpful Engaging

Oscar H.

May 2, 2023

At the moment, I don't delete any of my subs or calibration frames. I save all of it. But later I will only keep the final image because the rest will take up too much space.

Nick Grundy

Topic starter

May 2, 2023

For the last 6 months I generated about 1.8TB worth of data (light, cal, frames mostly). I upgraded more recently to the asi6200 which doubled the size of subs, and started doing more planetary/lunar. I didn't get crazy on the math yet, but I'm estimating 3-4TB of new data over the next 6-7 months. That's not a small number anymore. No one else is having this issue yet?

If we have the calibrated light frames saved already, how come we wouldn't delete the flats? Is there something that happens in calibration that will differ later? I've noticed that saving the calibrated frames in 16-bit integer files from Astro Pixel is the same size as the raw subs. (so no major benefit in storage) Technically is there some data lost if I save them in 16-bit integer? It's the same bit rate as the original sub if I understand it correctly. Does anyone know for sure?

Engaging

Doug Summers

May 2, 2023

If you're saving the calibrated light frames, then there would be no need to save the flats, but the question is why do that as opposed to saving the master flat, master dark, and just the raw fits files? The storage requirements would be much reduced (at the cost of needing to reprocess if/when necessary).

Well Written Insightful Respectful Concise Engaging

Nick Grundy

Topic starter

May 2, 2023

Doug Summers:
If you're saving the calibrated light frames, then there would be no need to save the flats, but the question is why do that as opposed to saving the master flat, master dark, and just the raw fits files? The storage requirements would be much reduced (at the cost of needing to reprocess if/when necessary).

at least according to what i've seen (based on ASI6200 fits files), the raw sub is around 120mb. A calibrated sub can be saved in various formats (32bit float, 32bit integer, 16bit integer, 8bit integer).

Actually, I just found the answer in APP forum. Apparently, I would lose some data if saving the calibrated frames at 16bit instead of 32bit. So it's clearly easiest to store the raw lights + cal masters I think.

Now I just have to plan for more storage

Related discussions

Flat Calibration Problem

Hi everyone, I'm having an issue with flats/calibration in my setup. My setup consists of a Carbon TS Newton 200/800 with Starizona Nexus (effectively 200/600), QHY ImageTrain (QHY 268M with QHY CFW), and AstroWanderer Rotator. A screenshot from ...

Discusses calibration frame management and flat frame retention strategies.

Mar 11, 2025

Two Questions Regarding Pre-Processing in Pixinsight

I made two problems for myself: 1. I didn't make my bias frames the same temperature as my lights, the lights are at -10c and the bias frames are at 0c. Darks are same temperature as lights. Is there anything I can do to compensate for this other...

Addresses bias frame storage decisions relevant to calibration data archival.

Dec 2, 2024

Trouble with Deep Sky Stacker applying flats correctly

I'm having trouble with stacking photos lately and not sure what changed. I'm seeing dust mote artifacts in stacked & calibrated images that shouldn't be there. Below is an image straight out of DSS. No further editing, STF applied. Y...

Concerns flat frame application; relevant to calibration frame retention decisions.

Dec 9, 2024

Saving TIFFs as JPGs

I am a newbie at all this, so forgive the basic question.. I have been shooting in RAW on a DSLR and stacking in DSS. This gives me a TIFF which I then process in Photoshop. But I can't then save that TIFF as a JPG (to post here, for example) bec...

Data retention strategies for RAW files and processing workflows.

Jun 2, 2021

How much data do I retain?

Related discussions