How to back up?

9 replies464 views
Mossyback avatar
If I start processing an image (galaxy/nebula) and save it and then at some point decide that I’m not happy with it, I have to start over. I normally begin again and save it with the original name and add a 1 or 2 etc. to the file name. There has to be a better approach to this. How do I back up to a previous point  in my processing of the image and then try something different?

Suggestions?

Hank
Well written Engaging
firstLight avatar
What you request would require an (A)IP software that allows saving a 'scene'. By scene I think of a feature to save the current work's state together with the history of what was done so far. Ideally, we would then be able to re-open an (A)IP project after weeks ... or month again to try a new approach.

The tricky and possibly unsolvable requirement would be the history part: How could an (A)IP software store all steps / layers done so far, which happen to reside in RAM, cache and/or swapspace .... including the software specific features to allow navigating, undoing and redoing whatever ... after the first re-open of that project in weeks or months?

~ * ~

Example: Siril

I really love the many realtime options for image processing Siril offers: I can try out and revoke, undo and redo everything from 'open file' to the current state. But when I left Siril there is no way anymore to re-open that file again ... and go back in history for some (or more) steps.

The best (most defensive) I can do is to save the very first steps (crop + stretch + photometry) as 'file-v1.fit' for this(!) state and than continure tweak other things and save this state as 'file-v2.fit' and so on. This is similar to what you do already, I guess.

I would not (more or less) post process an image to it's end – until it meets my expectations – and then(!) save it as 'file-v1.fit' , because there is no option to go back in history. Thus I save a 'history version' at some point I am accustomed to.

I believe, there is no other way in Siril: Saving intermittend processing steps to which I can return later, when needen plus keeping the logfile so I at least can read what I did before, when I re-open such a project to rework it.

~ * ~

Example: Gimp

Gimp has layers. I developed an individual naming convention for my layers and layer groups. It's a matter of discipline to follow this individually thought out naming convention. After a long break from this one project, I re-opened it for the first time again and was very happy about my layers naming convention which gave good hints and/or precise parameters and values for what I had done at that past time.

Then I can switch off the visibiliy of one or more layers, copy them to a new layer version and start over. Because of my strictly hierarchical ordered layers, I am able to rather quickly re-catch visually what I had in mind 'then' and what I did[i].[/i]

It works for me, tough I only occasionaly use Gimp, e.g. in the context of star removal (StarNet) and than getting them back in.
Helpful
Steve Solon avatar
Hi Hank,
When I process in PixInsight, for example, I do the calibration, cosmetic correction, and alignment of all the subframes, them integrate them into the single image. I save this image as a “Raw xxx” before I do any post- processing. That way, all the initial work is saved in case I’m not happy with a processing outcome. I do this for various steps in the post-process train- save the image at points along the way. 

I usually end up with 2-4 “working images”. When I’m finally happy with all the post-processing results, I save that image as a “Master xxx”. At that point, I’m comfortable deleting all the single working images. 

Finally, I always save the folder containing all the raw subs, as well as the folder containing all the aligned subs. The folders containing the calibrated raws, cosmetically-corrected raws, and debayerd raws (if applicable) are emptied to retain memory space on the external hard drive where I save everything. 

I hope this answers some of your query. If not, shoot more questions to : galaxyshots84[email]galaxyshots84@gmail.com[/email]. 

 - - Steve
Helpful Respectful Engaging Supportive
dkamen avatar
On Pixinsight you can simply save the project.

That includes every image you have open with its full processing history and every process you have instantiated as an icon on a desktop.

The process history is a very powerful tool in its own right. If you have applied 10 things, you can clone the image (to save it for reference) and then revert to e.g. step 6. Or you can apply the same process history to a different original image, to compare the results.


When I know I will be working on something for weeks, I use a project.
Well written Helpful Insightful Concise
firstLight avatar
Yes, PixInsight can do this!

<off topic>
I currently have it's trial license but a too tedious progress. Frustrating: Every single step I followed with several beginner videos on YouTube yielded issue on my installations, e.g. undocumented locations where some settings are to be made. Or increasing / improving the swap by adding directories on faster SSD drive. Set them up and after the next start they are gone again, with '/tmp' being again the single swap location.

I cannot solve an image the recommended way because in my installation the astrometry dialog looks smaller and with less input fields  than shown in the 'tutorial' .... in particular, no field is available to enter the object name so PI knows what object / region it shall search. Very strange ... and so time consuming, just for the basics.

... and more
</off topic>
Alex Ranous avatar
As mentioned earlier, Pixinsight projects are very handy to keep the history of what you did.  When I'm actively working on a project, I always use projects and save them periodically.  This also lets me switch between different projects as I'm often working on more than one target at a time. 

One problem with projects, is that they're fairly inefficient with their data storage.  Even though PI uses compression, it basically saves a copy of each revision step in a given image.  If you do lots of tweaks like I do, it can really start to add up. 

The process I've ended up with is that when I'm "done" with an image, I "flatten" the individual image windows by cloning them and deleting the original image.  The cloned images keep a record of all the steps performed to create the image, but you loose the ability to roll back to an earlier state.  I make sure I keep copies of everything that went into creating the individual images like all the masks and whatnot.  This lets me at some future point to go back and recreate things if I need to.  I then save the original subs, the generated masters, and the project file onto my NAS for archiving.

If I revisit an image after some time, I'm usually adding data or taking advantage of improved pre-processing enhancements, so rolling back isn't needed, but I do want a record of the steps I took to create the original images.
Helpful
Mossyback avatar
I appreciate you folks trying to help. 

As an example, suppose I use DBE and don't like the result and I want to go back and try something else. Do I go back using History and if so how do I do that?

The way that I've been doing it is (don't save and exit PI and then re-open the project) just seems extreme. I'm obviously not understanding.
Frank Alvaro avatar
As a relative newbie I still can't get the hang of using projects, so what I do is:

- open the original integrated image and immediately clone it. Close the original and rename the clone "Master"

- perform the first processing step, eg SPCC. When completed, rename the image identifier as "Master_SPCC", save the image as an xisf, produce a clone, then close the file. The clone's identifier will be "Master_SPCC_clone"

- perform the next processing step on that clone, eg DBE. When done, modify the identifier to "Master_SPCC_DBE", save as a new file, produce a new clone, then close the original image.

Continue for each step.

By doing this I have a saved version of each step, with the identifiers showing all the processing steps up to that image, eg "Master_SPCC_DBE_BlurX_GHS_NoiseX……" While this process doesn't show what parameters I used in each step, at least I can go back to any of the saved images, clone the clone, and try something new.

At the end when I have my final image I'll generally delete most of the saved copies, except for the early ones with SPCC, DBE and cropping done, as they're likely to be unchanged in any new processing.


Frank
Helpful
Rich DeMidio avatar
I also use PixInsight  and love the project capability.  You can also utilize the history so just evolve the master and if things go wrong, you simply revert back to the step that was good and restart.  For example, I mess up curves and saturation.  Just undo back to where it was with HT.   History is your friend so don't make it complicated with clones and such.  When you clone, you lose the history.  Also, I like to set the identifier early to reflect the target (i.e. Master_100x180). 

I also like to utilize workspaces in projects.  I name them typically with the date i.e. Jan10_Integration_100x180, and so forth.  T I use an external drive to store all the work.  Finally, I use IDrive for backing up my subs only.

I forgot to mention that I have a template project.  That project has all of my process icons and steps documenting my workflow.  When I am ready to process a new target,  I load  template project, then do a save project with appropriate name (i.e. M45).  I prefer a bundled project which needs to be checked since it is not the default.  This places all content in a single folder.

Hope this helps.

Rich
Helpful Engaging Supportive
Mossyback avatar
Thanks for all of the suggestions. Now I'll have to sort out something that will work for me.

Hank
Well written
Related discussions
How do you organize your lights, flats, biases and dark frames??
Hi folks, I need some suggestions.. per my subject line, how are you all organizing your lights and calibration frames?? I image with a ASIair… and generally I plug in a 16GB usb thumb drive into one of the USB ports for the night to store my i...
Discusses file organization methods relevant to managing multiple image processing versions.
Aug 11, 2024