Back to forums

How do you organize your lights, flats, biases and dark frames??

Processing Calibration Acquisition Deep Sky
8 replies•326 views
peng155 avatar
peng155 avatar
Hi folks,

I need some suggestions.. per my subject line, how are you all organizing your lights and calibration frames??

I image with a ASIair… and generally I plug in a 16GB usb thumb drive into one of the USB ports for the night to store my images.

When the session is over, or the next day, I'll remove the thumb drive and transfer all of the frames to a external 500GB SSD drive. Orginally I had a parent folder for each object, and within that parent folder, I had my sub-folders for the lights, darks biases and flats.

When I first begain playing around imaging DSO's I normally used the same gain, and exposure time setings, and to reduce the amount of space i was using on the SSD, I paired down the folders to just lights, and had another parent folder called Master Calibration, and called that good. But now that I've began to experiment with different exposure time and gain settings… I'm getting confused as to which calibration files I need to use with which lights.

One of the things I'm going to need to do is start a notebook for my imaging sessions and write down EVERYTHING I do with regards to changes to the imaging rig for the session…. But as far as my imaging frames go, I'm considering going back to the Parent folder for each object I image, and have sub-folders for all of my lights, and calibration frames.. even thou this will use up more drive space with duplicate calibration frames.. at least if I want to re-stack or color process an object later with a new technique… I'm not left scratching my head trying to remeber what calibration frames I used.

For you folks that have been doing this for a while, how are you organizing your files??

Thanks
Helpful Engaging
Miguel T. avatar
Miguel T. avatar
Darks:
/Calibration/Darks/<temperature>/<gain>/MasterDark_<exposure>.xisf

Flats:
/Calibration/Flats/<yyyyMMdd>/MasterFlat_<yyyyMMddsmilefilter>.xisf

Lights:
/Projects/<target>/<filter>/


If I had two target in one night due to the first one going below horizon mid-night then the flat taken is good for both targets. This is why I keep them separate and referenced by date taken. Don't need to keep the flats forever. Once you're done processing your target you can keep the calibrated lights and dump the raw lights and associated flats. Later, if you add more data you only need to calibrate your new data and include the already calibrated ones when stacking.
John Hayes avatar
John Hayes avatar
Phil,
My favorite tool in PI that goes a long way to sorting out the kind of problem you outlined is the FITSkeyword script.  You can throw a huge pile of data at it that contains different exposures, filters, temperatures, binning, … whatever and it will sort it all out and neatly organize everything by file folders.  For me, it is a huge labor saver.

If you don't use PI, this isn't much help but maybe there's another similar tool you can find to do the same thing.

John
Well Written Helpful Concise Engaging Supportive
peng155 avatar
peng155 avatar
Miguel T.:
Darks:
/Calibration/Darks/<temperature>/<gain>/MasterDark_<exposure>.xisf

Flats:
/Calibration/Flats/<yyyyMMdd>/MasterFlat_<yyyyMMdd>_<filter>.xisf

Lights:
/Projects/<target>/<filter>/


If I had two target in one night due to the first one going below horizon mid-night then the flat taken is good for both targets. This is why I keep them separate and referenced by date taken. Don't need to keep the flats forever. Once you're done processing your target you can keep the calibrated lights and dump the raw lights and associated flats. Later, if you add more data you only need to calibrate your new data and include the already calibrated ones when stacking.

Miguel,

Thanks for the suggestion... I'll give this a look and see if it'll work for me, but from a quick glance it'll probably work with a few minor tweks
peng155 avatar
peng155 avatar
John Hayes:
Phil,
My favorite tool in PI that goes a long way to sorting out the kind of problem you outlined is the FITSkeyword script.  You can throw a huge pile of data at it that contains different exposures, filters, temperatures, binning, ... whatever and it will sort it all out and neatly organize everything by file folders.  For me, it is a huge labor saver.

If you don't use PI, this isn't much help but maybe there's another similar tool you can find to do the same thing.

John

Hi John,

Thanks for the suggestion... Unfortunetly I'm currently doing all of my image processing on a M2 Mac, since I've just begun learning about Image processing... I'm using the free softare Siril, and GIMP. At least for me right now it a cost effect way for me to do all of my learning about image processing, and color correction..

Probably in the future I'll most likely upgrade to PI, but until I figure all of the nuances of Image processing.... adding a new piece of software would probably just fustrate me... But thanks I'll keep this in mind
Ewen avatar
Ewen avatar
John Hayes:
Phil,
My favorite tool in PI that goes a long way to sorting out the kind of problem you outlined is the FITSkeyword script.  You can throw a huge pile of data at it that contains different exposures, filters, temperatures, binning, ... whatever and it will sort it all out and neatly organize everything by file folders.  For me, it is a huge labor saver.

If you don't use PI, this isn't much help but maybe there's another similar tool you can find to do the same thing.

John

Thank you for this! I had never heard of this script before. It looks like a huge time/labour saver.
Well Written Respectful
Quinn Groessl avatar
Quinn Groessl avatar
Using a cooled camera I just have a folder with my master dark and master bias in it. I update those maybe once a year or so. Then for lights and flats I just keep in separate folders. My folder structure is Camera\telescope\Object\date\exposuretype\filter\

So mine typically would look something like B:\2600mm\FRA300\M31\2024-08-11\light\Red\….
Helpful
Josh avatar
Josh avatar
Here's the layout of my folder structure. It's been incredibly robust and keeps everything very well organised, and has made my stacking and processing very painless. I take advantage of the keyword functionality in the WBPP script in PixInsight, not sure if Siril has similar, but this kind of strucutre should work regardless.

D:\ 01. Calibration \ Camera \ Year \ DARK   (I also name each file with Gain, binning, set-temp and exposure time)
D:\ 01. Calibration \ Camera \ Year \ BIAS (as above)

D:\ Target \ YR_Cam_Scope_Rotation
  • (eg. D:\ M42 \ 23_2600_106_R145.00 [b]\[/b] (this tells me the year, the camera, the scope and the rotation of this imaging run, this way i can keep all of my imaging grouped by target, and can easily replicate to add new data to any set I wish)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT
  • (all my files required for integration)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ FLAT \ S_#
  • (I use the "S_#" keyword, where "#" is an incremental number, so i can easily match the flats with the lights, very handy because I usually go a few months on the same flats. When i take a new set of flats, I create a new folder in the FLAT and LIGHT folders and increment the # number.)

D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ LIGHT \ S_#   (as above)
D:\ Target \ YR_Cam_Scope_rotation \ 01. INT \ LIGHT \ DateD:\ Target \ YR_Cam_Scope_rotation \ 02.STK   (WBPP/Stacking output)

D:\ Target \ YR_Cam_Scope_rotation \ 03. PRC   (all processing tasks/intermediary files/pixinsight project files)

Here is the folder structure in practice for a Mosaic.
Helpful
Charles Hagen avatar
Charles Hagen avatar
Perhaps this is a good time to introduce Lumidex, it is an open source and free astro-image file management program, it reads in header data from .FITS and .XISF files. Everything is loaded into a searchable database allowing you to filter your data and keep track of it in a file structure agnostic way. This in my opinion is the most ideal way to work with your files, it massively simplifies searching for and locating raw and calibration data, even if its spread out over many different locations. We're working on bringing more features to it and would love more community feedback if you're interested. Its very straight forward to get set up and using it and there are tons of ways to filter and sort your data.
Helpful Engaging