Managing your aquisition data - Forums

Topic starter

Jun 11, 2016

I am curious to know what others use as their system for handling and in particular, storage of Raw data files.

My process uses Pixinsight as my primary process software, however as part of the process PixInsight will create a folder of format converted and registered images which greatly increases the storage requirements.

So I guess my question is whether you retain the original RAW files as taken on the camera as well as the .XISF files generated by PixInsight, or do you delete the RAW afterwards?

As a side note, using any form of cloud type storage is not an option for me here in Australia, as our internet quality is crap and in particular, our upload speeds make a 3rd party cloud option un-workable

Flint

Engaging

Goofi

Jun 11, 2016

What I do might be a little goofi, but it works for me. Hard drives are pretty cheap, so I have a lot of hard drive space to start with.

I run PI and let it do its thing, saving everything. Then, once a month I backup everything onto an external USB hard drive.
After they're backed up, I prune the registered files out (I can always re-register them), but keep the lights (FITS off the camera) and calibrated files.
I also backup my calibration files - just in case. Finally, any working/temp files I've created usually get deleted unless there's a reason I need them.

Helpful

Josh Smith

Jun 12, 2016

I've gone the route of getting a separate USB thumb drive for every image. It is nice because it keeps everything organized and is easy to go back to if you collect more data on a target. Also, nice to go back and see processing steps used to learn from later. Idea was gleaned from Adam Block's USB stick tutorials. It seemed great and worked well, so I copied it for my stuff.

Helpful Concise

Thilo Frey

Jun 13, 2016

I use a similar way like Chris. I have a 4 terra HDD in the PC as main storage. All rawdata (and other stuff) will be stored there. Processing results are stored and used on an 2nd SSD disk (the 1st SSD is the program and Windows disk), and only the end Result goes back to the storage disk. A Mirror of the storage disk is done on a NAS, transferred a few minutes after changing a file automaticly. The NAS also acts as a personal cloud, so i have access everywhere to all the data. A weekly full backup of the NAS to 2 offline HDD prevents data loss in case of viruses or similar things, because i think, the whole PC and NAS could be affected then. Every week to the other HDD. Ok, this sounds perhaps a little overcautious, but I only use electronical storage for all my data and photos since the last 30 years and loosing this stuff…. hmmm no

Helpful Engaging Supportive

Robert Q. Kimball

Jun 14, 2016

This is a good question.

I have a "generic" folder saved on my desktop. The folderstructure looks like this:

Generic Folder
Raw_Date
Lights
Darks
Bias
Flats
Flat_Darks
Integration Files
Calibbrated_Flats
Calibrated_Lights
Debayered_Lights
Registed_Lights
ImageProcessing_Files
Master_Files
Every Time I collect data, I make a copy of this "generic" folder and name it like this:

16_4_23_RingNebula_LasCruces

When I finish with my processing, I copy the whole folder onto my external hard drive. It is huge so I don't bother culling files. I like to keep everything, because as I learn new processing tricks, I often go back and reprocess everything from scratch.

Bob

Helpful

Robert Q. Kimball

Jun 14, 2016

Helpful

Wei-Hao Wang

Perfect post

Jun 15, 2016

I never delete raw images. They allow you to reprocess your images once your processing skill improves or once there is some brand new way of dealing with the data. Never delete the raw images.

On the other hand, I always delete the intermediate files generated by PixInsight once I stack them and feel satisfied about the stacked image. Of course, it will take some time to process the stacked image to make sure it is good. So I may leave the intermediate files there for a few days, but I never want them to stay in my HD permanently. If for some odd reason that they are needed later, they can always be re-generated from the raw files. Not a big deal.

Well Written Helpful Insightful Concise

Bill Mark

Aug 13, 2016

same here ...
keep the source (raws) , stacked stuff to try different processing, and don't forget the final pictures. Until ASTRO bin I tended to lose the finals...

of course you can big NAS with a few Tera bytes save everything

Richard Pattie

Sep 21, 2016

Guess I use an old school approach. After a night of capturing, I immediately save the raw files to DVD, then do the same with supporting files (flats) as they're created. Darks are in a library on their own DVD. I don't save any of the processed files. I can usually get 3 targets' worth of data per DVD.

Well Written Concise

Related discussions

Astronomik Filter Halo Story

This will be long, TLDR: Myself and at least a few other competent imagers have had recent issues with Astronomik Deep Sky RGB filters and Astronomik's customer service. Buyer beware. I will preface this by stating that I am not looking to bash a...

Feb 8, 2025

Both posts seek advice or share experiences about practical challenges encountered in astronomical imaging workflows.