Pixinsight Files

I have been using Pixinsight for about a year now for processing It's amazing! What a game changer. I recently began using it for Preprocessing and I'm curious after all is said and done which files everyone keeps. I think its obvious to keep the masters it creates, but what about all of the other calibrated and registered images? These files are enormous! I do have three gigs of space for storage however, these three images have already used up app500 gigs of space. How does everyone treat these extra files? Save them into an external drive or just delete them?

I save original lights, master dark, master flat, master bias (or flat dark). Then I delete the calibrated, cosmetic correction and registrated files (as you said those are what eat up the space). You can always reintegrate from there if wbpp has some new fancy process in the future, or want to add more data.

I also keep only the original lights/flats/darks/biases, and remove calibrated, registered, debayered, as they can be regenerated should the need arise. But I find it important to create a file with a good set of notes on how the files were calibrated (what darks, biases, etc.), registered (what reference frame), etc. – otherwise I'm sure I would forget soon enough.

i only keep the masters lights, darks, flats, dark flats

I used to store lights/flats/dark but after 3 years doing that I only keep calibrated lights, if I need to integrate more data it is easier than having to recalibrate everything again.

I discard flats, registered frames etc…
I also keep my master dark for an year as I reuse it for new images (I have a cooled camera)

I also save the masters and lights and delete the calibrated and registered files. I use a library of flats and darks, which I can usually use for 3 to 6 months because I have a fixed setup.

I keep the calibrated folder. That's it.

No need for flats/bias/lights when you have all that data in the 'calibrated' image already. The registered folder is the same as the calibrated, only the images are all aligned… so no need for that either. It really doesn't make any sense to delete calibrated data.

I'm sorry, I meant to say I have three Tb of space, not three Gb.

I do pretty much the same …

I keep the original lights filed by date, and master calibration (bias/dark/flat etc) files in a separate folder filed by date, because flats might be used for calibrating images of different objects captured during a single session, and bias/darks for calibrating images captured across many imaging sessions.

When I process lights, I keep the logs (eg WBPP logs), so I have a record of the calibration process, and master files in a folder filed by object. I delete all the calibration frames - as others have said, they can be regenerated later if needed.

I also save the processed image as a project - these can be quite large files, but they let me pick up my processing where I left off, and preserve the processing history.

And as Francesco has already mentioned, I keep a set of notes with each project so I know what I did and why.

This thread raises a good point about keeping calibrated lights. I have been keeping raw lights and master lights/darks/flats. If I never plan to cut new master darks or flats (i.e. don't keep the raw files) then why should I not just keep the calibrated lights? I've been keeping those master darks/flats for every profile+target+date to keep things simple (no symlinks or shortcuts). Looks like a bit over 1000 master darks (~270GB) and 1200 master flats (~350GB). It's adding up over time

Francesco Meschia:
I also keep only the original lights/flats/biases, and remove calibrated, registered, debayered, as they can be regenerated should the need arise. But I find it important to create a file with a good set of notes on how the files were calibrated (what darks, biases, etc.), registered (what reference frame), etc. – otherwise I'm sure I would forget soon enough.

I do exactly the same like Francesco, removing calibrated and registered images (as they can be regenerated from the original lights/flats), but taking note in a specific file of the settings done in a way to be able to repeat the flow in case of need.

I follow a very similar scheme to Jeff Horn: retain all masters generated by WBPP including flats and flat-darks, and the original lights; delete everything else. WBPP/PI can regenrate the deleted files if required. I haven't deleted the LN reference and the uncropped calibrated masters, but they're on my list to clear space when it becomes an issue as these also can be regenerated.

I hope this helps, Paul.

Can I ask an open question here to those that have responded so far?: Why are you deleting calibrated data? And wanting to recalibrate it again later? Calibrated data should be good to go, you'll never need to mess with it again. You can restack the data anytime without having to use flats and bias. You can easily integrate it with other scopes and projects.

Brian Puhl:
Can I ask an open question here to those that have responded so far?: Why are you deleting calibrated data? And wanting to recalibrate it again later? Calibrated data should be good to go, you'll never need to mess with it again. You can restack the data anytime without having to use flats and bias. You can easily integrate it with other scopes and projects.

with OSC cameras each calibrated file (even prior to debayering) is still a good 5x or more larger than the raw image file. Just to save more space for storage management.

different ways to skin that cat obviously, here is my 5C

1) WBPP has in the recent past been a bit "difficult" - registration often fails- so I am not as trusting as 6 months ago that these processes are iron clad.
2) Calibration may change in the future, so I think it's critical to protect the original data.

I keep the (good) lights (16bit FITS), that alone saves a lot of space as calibrated and registered files are saved as 32bit XIFS's.
I keep bias darks and flats as fits files, in a general catalog, that's on average 6-10 flat-sets per year. biases and darks I will rerun typically once a year...

Brian,

For me it's simply a matter of priorities. When would I need to reuse my calibrated, cosmetically corrected, debayered, or registered individual frames again? - in nearly 2 years of using PI, my answer is never. If I should need to regenerate any of them, I would probably need to tweak some of WBPP's parameters, so the files I generated in my first run would become redundant, anyhow. Of much higher priority, for me, is disk space. With serveral hundred targets under your belt, it looms LARGE!

I hope this answers your question. Paul.

I only keep the unprocessed light frames, the master darks/flat-darks/biases/flats and obviously the final stacked up image for each imaging session (which might run for years at times). I don't see what use I would have for any other file generated by PI or whatever else I was using at the time. And I can go back quite a long time…

Great discussion! I kind of do the same thing except my original raw files are saved to a cloud storage and then I import them to my pc for processing. As a few folks already mentioned it's very important to keep a log, notes, something so you know what you did!
You can always save files on a flash drive until you are sure you are happy with the results. Once or twice I would see something, artifact or some other problem that I wasn't sure where it came from and had to go back to earlier stages of processing or to the original files.
HTH
Dave

Scott Stirling:
Brian Puhl:
Can I ask an open question here to those that have responded so far?: Why are you deleting calibrated data? And wanting to recalibrate it again later? Calibrated data should be good to go, you'll never need to mess with it again. You can restack the data anytime without having to use flats and bias. You can easily integrate it with other scopes and projects.

with OSC cameras each calibrated file (even prior to debayering) is still a good 5x or more larger than the raw image file. Just to save more space for storage management.

Okay this actually makes sense. I just checked some of my old OSC data. My calibrated lights are only twice the size though. /shrug Debayered is on check with what you're saying however.

Brian Puhl:
Can I ask an open question here to those that have responded so far?: Why are you deleting calibrated data? And wanting to recalibrate it again later? Calibrated data should be good to go, you'll never need to mess with it again. You can restack the data anytime without having to use flats and bias. You can easily integrate it with other scopes and projects.

For two reasons :-

1. First and foremost, sets of calibrated files can consume a HUGE amount of disk space,
particularly for some processes such as deBayering and drizzling full-frame images;

2. If I ever revisit an image, I'm almost certainly going to process it in a slightly different way, or use an updated version of the software,
in which case I wouldn't reuse the original calibrated frames anyway.

In terms of workflow, saving calibration frames to disk means they don't have to be stored in RAM, which reduces memory pressure, and increases processing performance. Some other image processing products (eg Affinity Photo) treat calibrated files as temporary and do not save them at all.

I hope that makes sense

Brian Puhl:
Scott Stirling:
Brian Puhl:
Can I ask an open question here to those that have responded so far?: Why are you deleting calibrated data? And wanting to recalibrate it again later? Calibrated data should be good to go, you'll never need to mess with it again. You can restack the data anytime without having to use flats and bias. You can easily integrate it with other scopes and projects.

with OSC cameras each calibrated file (even prior to debayering) is still a good 5x or more larger than the raw image file. Just to save more space for storage management.

Okay this actually makes sense. I just checked some of my old OSC data. My calibrated lights are only twice the size though. /shrug Debayered is on check with what you're saying however.

Another for comparison, Canon CR3 files from the EOS Ra come out raw around 20 MB, then the calibrated XISF output is around 100 MB or more usually. The debayered files are around 325 MB.

I sometimes keep my debayered frames after WBPP runs, just in case I want to run Sumframe Selector manually and then re-do registration, normalization (if applicable) and integration. Beyond that, like many others, I keep masters and original lights.

I no longer keep darks, as Cuiv the Lazy Geek persuaded me that they are not necessary for my camera. See https://youtu.be/0wB5PS3pGxA?feature=shared

Wishing every clear skies!

Jeff

I use a Network Attached Storage (NAS) configured as a Redundant Array of Independent/Inexpensive Disks (RAID). This provides terabytes of storage. I download data directly from the camera/capture computer to the NAS and it is immediately available to any computer on my network. If a disk fails it can be replaced by swapping in a new disk and the system will configure it with no data loss. The capacity can be grown to hundreds of terabytes by swapping in larger drives. I don’t waste time managing files or moving data from one computer to another. When a new capability comes along like NormalizedScaleGradient, I don’t have to reconstruct registration files to take advantage of it.

I shoot biases and darks once every 6 months. I keep these files in calibration/533mc/2023q4/bias and …/darks_gain100_-10C. The first time I ran these through I save the generated masters as well, that’s for ease/speed in the future. The lights I store per year and (main) subject. So _2024/m81/lights. If shoot in more than one day I add the ISO date to that (f.e. 2024-01-19). There’s _2024/m81/flats. If I shot more than one target with the same flats I add a readme.txt with a note to where the flats actually are. Now I do not store the master lights, because when I feel the urge to re-process an image I rerun WPBB, so I can benefit from the advancements made there. I’m reconsidering this. Of all these files I have at least 2 copies, one on OneDrive, one on my NAS. When a year has gone by I zip everything and upload the zip to AWS Glacier.
After typing all this I realize that the OP didn’t ask about where one stores his/her files, but hey, this is just my 2c.

I keep the calibrated files along with the final master outputs that have been created. Also, don't forget to save your master flats, master darks, and master biases, since they need to be matched with the calibrated lights (but you probably already know that).

Related discussions