Storage and Computers

Great question, and one I've been asking myself too as my data drive is almost at its limits.

Personally, I am backing up all my captures to an external drive and removing them from my main processing rig.  I'm keeping calibrated frames that I have selected via PixInsight's SubFrameSelector, but discarding the raw images, the registered, calibrated, measured directories etc..

I have a 2TB NVme drive for my main processing data drive and it is always getting full if I don't archive data/clean-up temporary files.

Data is precious - in particular if you consider what amount of time you invest getting and processing it. So saving on data storage is the wrong spot in my opinion. After all we are doing all of this for the data and the final image. One of the unique features of our hobby is that every year you can collect additional data on the same subject, so taking good care about your data is crucial in my opinion.

I keep 3 copies of my data:

• The original data
• A local backup on my RAID 10 NAS
• A remote backup on my CrashPlan service

So even if the house burns down at least I still have my data ...

With respect to astrophotography I save

• My dark library
• The various configuration files of my imaging software
• My Lightroom catalogue and image library
• For each session:
• The Lights
• The Flats
• The Dark Flats
• The master dark used for calibration - this is redundant but this way I always have the proper master dark in place for my data in case I ever want to reprocess.
• The final integrated channels
• The PixInsight bundled project file with the final image including the complete processing history
• The 16 bit TIFF file with all layers in case I did some retouching in Photoshop
• The PNG files derived from the TIFF file in various sizes for publication or as screen background imported and keyworded in Lightroom
• The NINA sequence
• My session plan and my session notes
• Currently: my phd logs since I'm new at using phd, I don't intend to store them permanently though.

I don't store intermediate files created by PI in the process of image integration like cosmetic corrected or registered images. I would be able to recreate them if necessary. 

Data is growing fast, my QHY 268m creates 50 MB fits raw files, so 20 GB of Data for one object is no exception. With my 8 TB SSD for storage in my PC plus additional 4 TB NVMe SSD for processing (about 50% free each) I still have some leeway though. NAS can get extended without issues anyway.

Clear skies
Wolfgang

Cold storage is relatively cheap but want some redundancy so a NAS was the solution and then the first NAS filled up so another bigger one added.

CS Sean

Great thread! Out of curiosity: You are not considering cloud storage? I use OneDrive only, and upload all projects - lights, darks, flats, bells'n whistles - there, after discarding bad subs, creating the final image, etc. Then again I'm fairly new to this, so I may have missed out on some key considerations among those in the know.

My third copy using CrashPlan by Code 24 is a cloud backup solution. Obviously you need a good internet connection to cope with the data volume. Nothing wrong with using cloud storage.

Clear skies
Wolfgang

Hi everyone,

In my case I use 2 external HDD (not SSD nor NVMe) and I use the SSD from the laptop just for the OS, the applications, and some documents and of course I have a folder in where I have the files that I will stack. In the external HDD there are files for storage purpose only.

Clear skies!

Daniel.

I forgot to say that my laptops are not the best option. 

The older one is a MackBookPro i5 4 cores 16GB RAM from 2012
The newer is a Dell Windows 11 i7 4 cores 16 GB RAM from 2021

But I think that the best option will be a desk computer, something powerful like a pc gaming or something like that. It would be fun to do a poll to find out the trend of colleagues with their equipment. Let's do it.

Here you are: https://www.astrobin.com/forum/c/astrophotography/other/do-you-use-a-laptor-or-a-desk-computer-for-satacking-and-processing/

Clear skies!

Daniel.

Its interesting to see all the different configs people use. I knew immediately when I got into this hobby that data storage would be kind of a big deal. I have a minor IT background and built my own systems since 1995. Going into this I hadn't built I system in nearly 10 years so I had a bit of a learning curve as far is new tech goes, particularly on the storage side if things. My 10 year old system was a 3rd gen I7 quad@ 3.2GHZ with 32gb DDR1600. It was designed originally for gaming and had dual GTX 660ti's. It had a total of 750gb SSD's on 2 drives along with another 4tb in SATA HDs and an additional 1tb Seagate portable. This system was fine for my initial DSLR processing with DSS, GIMP and Affinity photo. When I purchased my GEM45 and ASI2600 I decided to step into Pixinsight and really felt it was time to modernize. I purchased a Clevo laptop(bare bones laptop) with 11th Gen I7 running Ubuntu Linux exclusively with 2tb NVME. This is how it benches PI:

CPU Identification
CPU vendor ............. GenuineIntel
CPU model .............. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 

System Information
Platform ............... Linux / Ubuntu 22.04.1 LTS
Operating system ....... 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022
GNU/LinuxCore version ........... PixInsight Core 1.8.9-1 (x64)Logical processors ..... 8
Total memory size ...... 31.136 GiB 

Execution Times
Total time ............. 01:24.97
CPU time ............... 01:18.00
Swap time .............. 00:06.95
Swap transfer rate ..... 2385.722 MiB/s 

Performance Indices
Total performance ...... 5536
CPU performance ........ 4853
Swap performance ....... 13213

This is the system I have been processing PI with for the last 8 months. If I was a little more motivated (or fully retired) I could probably improve the performance by tweaking the swap system, but my I only have so much learning capacity at my age. I added a 5TB Seagate portable HD for backup. I am still using an inexpensive laptop outdoors for capture and guiding but plan on moving to Astroberry at some point (already have my box, but need to learn KStars/EKOS before the full leap), but in the meantime move files with a 1TB Sandisk Extreme portable SSD. 

Additionally, I just recently built an I9, 64GB DDR4-4400, 2TB NVME, Radeon 6900xt desktop that I am currently running Win11 on, primarily as a gaming system. But I got the itch to try it out for my astro endeavors and have a 2nd NVME drive on the way to dual boot Linux (not sure if I will stick with Ubuntu distro). I plan to test out PI with the GPU processing techniques I have briefly read about.

I probably got away fro the OP question a bit. So far I tend to store everything, lights, calibration frames. I have started building a dark library, which will save some space in the future.  But for the most part, I delete nothing but junk lights like focus problems, guiding farts, Elon Musk's photo bombs, and so on. If there is any hill I will die on as far as recommendations, I would encourage saving money and adding stability/speed/support by shifting to a Linux platform. The modern GUIs make it a pretty smooth transition from Windows/Mac. I dabbled with Unix based systems in the late 90's/early 2000's and it is so much easier to work with for the hobbyist plebe like myself now than back then.

Jay Stanley:
Thanks all, this is some great information.  I'm not a total computer idiot but I can do just enought to have to call a tech Chime in on this, I'm looking at a Dell with Win11 (I klnow my Apple freinds here may be shaking their head)  I have other software to run that won't on an Apple. Here,s the specs:

Dell Inspiron 3910 Business Desktop Computer I 12th Gen Intel 6-core i5-12400 Processor 
32GB DDR4 
2TB SSD + 2TB HDD 
Intel UHD Graphics 730  
DisplayPort HDMI 
DVD-RW 
Bluetooth 
Win11 

and then add a USB 6T backup.

The technology changes so rapidly now that I don't think I could ever be on the "cutting edge" but it's time to retire the old Dell Optiplex 780 circa 2009.
I'd love to get Pixinsight but my old machine can't run that software. 

Guess I should go back to my old Fortran programming days

CS
Zuukz

The main question when evaluating a PC is always: What is your main use case for it?
In case it's astrophotography image processing I would recommend at least an i7 cpu and some more RAM. The i5 with 32 GB will do the job too, but the performance boost of an i7 and more RAM will be noticable if budget allows.

Clear skies
Wolfgang

Everything else being equal the price difference should not be that much. You may check the AMD Ryzen 7 options too. Starting with an I7/Ryzen 7 with 32 GB is another sensible option. You just have to check with the dealer that there are still free RAM slots available for a later RAM upgrade (this means you need 2x16 GB RAM instead of 4x8GB).

However, as I already pointed out, the i5 with 32 GB will do the job too, just somewhat slower. It will significantly outperform your old setup. Even with the i5 I would check whether a future RAM upgrade is possible. 

Clear skies
Wolfgang

I keep all of my 'lights' and 'flats' on a 4TB SDD. 

I have a library of darks and biases on another 4 TB disk.

The OS and all programs are on a VME drive, and all other personal data is on a separate SSD.

All of this gets backup up to a 12 TB external drive.

Roger

The offline backup already protects you against everything except theft and influence of the elements. So while there is some gap in your protection with a one day delay of an online backup, for private data it's acceptably low. Probability that the event strikes exactly while you're transferring data is low if you don't live in Chile with 300 nights of clear skies and data. In the worst case you're losing (part of) your last  session which is bad but not catastrophic for amateurs. For a business obiously other rules and risk assessments apply. 

Clear skies
Wolfgang

I keep lights and calibration frames on external hard drives. Calibration frames are masters only since I reuse the dark frames after the first integration run and the flat and flat darks until something changes with the optical train. Those are kept with the individual projects that use them. 

The raw uncalibrated lights are kept because as the image processing software continues to evolve there is always a reason to go back and reprocess.

My imaging computer is an AMD with 32 cores,128GB RAM and 2TB NVME drives. The extra power is needed given the size of the frames.