Multi-session organization - Forums

Topic starter

Mar 26, 2023

One of the tools we have in our high-Bortle bag-o-tricks is integrating data from multiple nights, sometimes months or years apart. During those intervals, our post-processing skills often advance/adapt/shift and it makes sense to restart those processes from scratch.

Over the last two years I’ve tried, and discarded, multiple data organization techniques in my effort to create a system that will maintain a level of organization over time and not completely fill my NAS which my NUC synchs the files to real-time.

My latest iteration involves using NINAs built-in system of saving by date and relying on a spreadsheet with a tab for each target that logs which days I captured that target, which camera/scope, how many of each filter/flats/dark flats.

I am currently maintaining all my raw capture files (not sure now long I’ll be able to do that) in the NINA-created folders within a “NUCcapture” folder. Then my Pixinsight WBPP output files are put into folders by target. I’m hoping this will allow me to more-easily find/track/re-integrate older/historical data later. I’m considering making a separate WBPP output file for each capture date under each target folder to organize the masters better… but I’m on the fence.

What techniques/folder-structures are other folks out there using?

Well Written Helpful Engaging

Elmiko

Mar 26, 2023

I save my data on 2- 4 tb external hard drive. Have about 5 of them.

James Peirce

Mar 26, 2023 · Edited: Mar 26, 2023

I sort my sessions by /Target/YYYY-MM-DD (Basic Equipment)/[Session Files]. "Basic Equipment" is something like "2022-11-09 (106 0.72x 2600m SHO) SV" which just records the date (in that format for sorting) and basic acquisition details about my equipment and an abbreviation for the imaging location—what I need to tell, at a glance, what the session may contain. I retain my source files and any integrations or masters associated with images I processed. The source files are compressed into archives, significantly cutting their storage size. I also have a directory for calibration frames where I may store flats used across sessions, for which masters are stored in the session files and the directories are referenced.

Just keeps things simple. If I want to reference work I have done with a target, I just use the target’s directory. It doesn’t need to be too rigid this way. I can use an alias if it is appropriate, for example.

I use the same archival structure on my NAS and on my editing drive. Except the NAS is where the source files and archived sessions are stored while the editing drive is focused on what I’m working with. If I wasn’t using a NAS for this, it would be a basic spinning-platter hard drive (or drives, designated by year(s) if multiple were necessary) backed up by Backblaze for redundancy, but I like to have everything at hand, and I have accumulated a pretty chonky amount of data, so I use a NAS.

A couple automated scripts serve to retrieve files from, say, a directory on my thumb drive, rename them, and sort them where they need to be going. I’m on a Mac. I use the programs Hazel and Automator (Apple’s app) with some built-in tools along with a sprinkle of JavaScript and a couple shell scripts to sort those details. But this is all just to speed up some busy work and not really related to the file structure above.

Helpful

Timothy Martin

IOTD Staff

Mar 26, 2023

Very similar. Spreadsheet, slight variation of what NINA creates directory-wise, and storage on the NAS. I have an 80TB NAS that's almost full. Just ordered an 80TB add-on unit. I'll probably have to add yet another 80TB unit soon. I'm leaving my Bortle 8 behind. I'm not moving, but I'm going to use my small refractor for road trips and convert my C11 to purely visual and planetary use. I'm building out three new scopes--a TOA130, an FSQ106, and a CDK12.5--that I'm going to plant at Deep Sky West. After four years of learning in my back yard under horrific sky glow, I'm ready to take the next step.

But back to your subject. I use APP for stacking, so all I have at the end of the process are the master darks, flats, and lights. I find I get much better results with APP than with WBPP with my light-polluted data. I crank up the local normalization correction to second degree with 10 iterations and it really helps reduce and smooth the gradients so that DBE is more effective.

Here's a typical directory structure for an image of Sh2-284 I shot across seven nights. I'll generally stack it several times (hence, directories like APP-s01-s05 indicating it's a stack for the first five sessions). Sometimes I may shoot multiple sets of flats (I hope not to, though), so in this one there are two directories for flats (I usually only keep the masters and discard the flat subs). There's a directory for dark masters of each duration I will need for the shot. And finally, there's a directory for each session, with subdirectories for each filter under that.

Here's what the spreadsheet looks like. Pretty simple. The appearance of "Daddy" in the filename indicates the scope. I've named the four scopes I currently have Baby (WO Z61), Mama (WO GS61), Daddy (C11), and Grandad (12" Apertura Dob, which was my first scope).

I save all my PixInsight projects to the NAS. Once I've finished an image, I create a "Capture" subdirectory under the project and move this entire directory structure there. Each PixInsight project is named with the official designation, colloquial name, scope, reducer or not, camera, year, and month. So, for example, for this project, the project name is "Sh2-284-Finger-Daddy-Reducer-6200MM-2023-03." Makes it easy to find later. The spreadsheet makes it easy to enter the detailed data here on Astrobin--especially with the new page Salvatore created for that. That's my story and I'm sticking to it. Hope it's of some use to you.

Engaging Supportive

noon

Topic starter

Mar 27, 2023 · Edited: Mar 27, 2023

Something literally JUST occurred to me as I was setting up a WBPP to run on last nights' files:

In the lower right corner of the dialog, where you can select either an automatic or manually selected file to register from… I just realized that I could save myself a lot of time and energy if I manually select the same registration file for every target… for example… if I shoot RGB on Monday, Tuesday, and Wednesday… I process Monday's data and output to "Target"/"Monday" folder…

Then Tuesday, I manually select Monday's "Registered" Red subframe 1 ( or other suitable frame) and process, pointing the output to a "Target"/"Tuesday" folder… all my registered subframes are now already registered with Monday's data.

Then Wednesday, I manually select Monday's "Registered" Red subframe 1 (e.g. same subframe I selected previously) and process, pointing the output to a "Target"/"Wednesday" folder…. all those registered subframes are now already registered with Monday and Tuesday's data…

Now I can merely run Image Integration with the data from all three days without RE-registering it all together… like I have been all these years.

This may be how everyone else has always done it and I'm just late to the dance… but I was today years old when I finally figured it out. Hope it helps someone.

The added benefit is that you can then keep only the registered files, which are already cosmetically corrected/normalized/drizzled, discard the previous intermediary files, and archive the original capture data (if desired).

Helpful

Mark Germani

Mar 27, 2023

Hi there:

Whenever I image, I use the following folder structure:

YYYY-MM-DD Target Name
Lights
Darks
Flats
Dark Flats
Rejects

When I process, I move these folders into a main folder for that target. After processing, I move each session out of the project folder and archive it separately with master (stacked) calibration frames. At this point, I delete rejected lights, and individual calibration frames, saving the masters. I don't save calibrated lights because I use a DSLR and calibrated FITS lights are WAY larger than uncalibrated RAW files (but this is likely to change in the near future). I save all the stacked project files in a separate folder, in case I want to revise them, but when I add more time to a target the following year, I usually drag in the archived session folders and reprocess from scratch.

One point to mention is that if I use the same darks or flats across multiple sessions (same temp, camera setup) I copy the calibration files into each archived session. I figure it's better than trying to figure it out when I try and use those sessions again, plus I keep the dates of the sessions that the master darks or flats apply to in the file name, so it's pretty easy to assign them appropriately when restacking later.

When I do add time to sessions, I don't stack each night and then stack those stacks, I load ALL of my lights and assign the appropriate calibration files and then stack with drizzle enabled. This means I can increase the scale or decrease the droplet size as I increase the number of lights over the years.

I keep a detailed spreadsheet to keep track of all of my sessions. Hope that's helpful!

CS,
Mark

Helpful