Long-term Data Storage

17 replies•658 views
Steven Fanutti avatar
Steven Fanutti avatar
Hi all,

As technology is always changing, I am curious to know how other AstroBin users are storing their precious photos. Aside from storing them on this site, do you use cloud storage, DVDs, photo development, etc.? Here's what I do:

Primary Electronic Storage: Backup hard drives. By far for me the preferred electronic method using both the Time Machine backup and manually using the tar command. Files are easily backed up and restored.

Secondary Electronic Storage: DVD. Yes, I still use this old technology to protect against data loss. Just in case I have to reset my computers and my files get corrupted, I can restore them from DVDs if necessary. However, DVDs don't last forever and in my experience are time-consuming to restore. Sometimes, I find it more efficient to tar archive my files and burn them to DVD and restore from a single archive later. If the project needs to be split across multiple DVDs, the split command works really well for that.

Physical Storage: Physical development. I develop any of my photos that I deemed worthy of printing. It can get expensive if you develop a lot of them, but if you worked really hard at getting photos just right, it is definitely worth it. The advantage of physical prints are that they are technology-proof. As long as you carefully handle them and store them properly, they will outlast all of your technology.

Also, I highly recommend purchasing a flatbed scanner. If you have printed photos and you accidentally delete the electronic files or just can't find the electronic versions, you can digitize the photo prints.

I don't really use cloud storage that much except to backup some photos that I might use for computer wallpaper. Service providers can always change their terms at will.

If I were to choose only one method, I would choose physical storage. No computer required to enjoy them.


​​​​​​That's how I store my images.
Quinn Groessl avatar
Quinn Groessl avatar
I have my completed photos in my google drive, dropbox, and my PC. Then all my individual exposures and all the other things are on my PC and my NAS. Unfortunately they are about 2 feet from each other, so it's only really protecting against drive failure, but I figure if I have a fire or some other catastrophic loss I'll have more important things to worry about.
Bill McLaughlin avatar
Bill McLaughlin avatar
My take on prints is that they are seldom worth the cost. Why? Simply because unlike terrestrial photos, they will be better next year given the rate of progress in hardware, software and probably your processing skills. Prints are too expensive to obsolete every year.


Just my .02
Steven Fanutti avatar
Steven Fanutti avatar
Bill McLaughlin:
My take on prints is that they are seldom worth the cost. Why? Simply because unlike terrestrial photos, they will be better next year given the rate of progress in hardware, software and probably your processing skills. Prints are too expensive to obsolete every year.


Just my .02


I understand your response. I have developed multiple versions of the same photo thinking that it was "final" only to learn new processing skills to make it better and redevelop it later on. For example:



Noise Reduction


​​
Thomas avatar
Thomas avatar
Primary is local external hard drive backups. I have a couple drives that get updated with important stuff periodically and then stored in the fire safe.

Secondary I use online cloud storage. I use iDrive and have found them reasonably priced and they will ship you a physical drive for backup or restoral if needed. I know way too many people that have lost everything that weren't using cloud storage as a backup should the worst happen.
Bill McLaughlin avatar
Bill McLaughlin avatar
Steven Fanutti:
Bill McLaughlin:
My take on prints is that they are seldom worth the cost. Why? Simply because unlike terrestrial photos, they will be better next year given the rate of progress in hardware, software and probably your processing skills. Prints are too expensive to obsolete every year.


Just my .02


I understand your response. I have developed multiple versions of the same photo thinking that it was "final" only to learn new processing skills to make it better and redevelop it later on. For example:



Noise Reduction


​​

One solution would be a large  electronic picture frame like that made by Samsung (aka "The Frame"). Expensive but can be kept up to date. I keep trying to convince myself to buy one. 
Benny Colyn avatar
Benny Colyn avatar
3-2-1 Backups are the gold standard for all digital data you want to keep:
  • 3 copies
  • 2 formats (for the purpose of this exercise, each cloud vendor is a different format since it has a different API from your PoV) to guard against one format becoming inaccessible
  • 1 of which is off-site (in case of a local disaster)


I have all my data (astro and other documents - most importantly the pictures of my kids growing up, family vacations, ...) on a primary NAS (ZFS-based), backed up to secondary NAS (previous machine, using scheduled ZFS send transfers, kept offline when not in use) and duplicated to Backblaze. Not what I would recommend for someone not working in IT, but there are more user-friendly options out there for doing pretty much the same.
Kyle Cerniglia avatar
Kyle Cerniglia avatar
If you're a server nerd, then LTO tape decks are an option. I'm currently getting one up and running now for my backups and astro data.
Mariusz Henn avatar
Mariusz Henn avatar
I use a NAS server on my home network with RAID 1 to ensure easy, reliable and fast access to data. Additionally, an automated monthly backup to Amazon S3 Glacier is done to prevent permanent data loss. With this configuration I feel quite comfortable.
Jens avatar
Jens avatar
I use a Nas as well which automatically backs all my data up to a Backup cloud storage which I can recover if something would break.
Works really well and I don't have to worry about having a ton of single external harddrives to go through.
Christian Großmann avatar
Christian Großmann avatar
I also use a NAS storage with HDDs. Raid is a must. It saved me so much trouble in the past. Exchanging dying HDDs is really simple and I never lost data for more than a decade now. I keep the data also on a local SSD in my laptop. This one I take with me nearly everywhere I go. This is what I process the images with. I keep a third backup on another USB drive, that I update about every three months, depending on the amount of new data.
Franco Grimoldi avatar
Franco Grimoldi avatar
I store everything on a NAS using RAID-5 for redundancy and then all is mirrored in Backblaze. It's the easiest/safest backup method I ever used: I just need to copy all the files/folders I care about into any of the NAS folders flagged for automated daily backups, then I forget about everything.

The NAS is only for storage, having a gigabit link, it's way too slow for any online processing, I do all the processing on a 2TB M.2 NVMe drive.
Sean Mc avatar
Sean Mc avatar
Online storage only for me. I have a 1.5 gigabit internet connection. Amazon has far better redundancy than I ever will.
Xenofon Nastos avatar
Xenofon Nastos avatar
Local hard drives (8TB) for the temporary storage and processing, a Software RAID5 (based on Linux) with local drives (12TB) for long term storage, final processed images (mainly 16bit TIF's) uploaded on my Google-cloud storage. I changed to this this three-step solution after almost loosing my complete recorded and processed data due to a WIN filesystem failure. There is no way around redundancy!
Ruediger avatar
Ruediger avatar
Hello all,

1. A HDD is no long term "backup"
HDD lose information after a couple of years (the denser the magnetic domains are, the faster), if the data is only read. If you want to store data for longer time on HDDs you have to copy the data every 3 to 5 years in order to refresh the magnetic domains.

2. A NAS is not a backup!
2.1 It is storage. Backups require long term stability and being safe from additional hardware failures e.,g. power surge. Over voltage in the PSU may kill all your data at once. Same does a firmware failure or a wrong click.
2.2 since it does not protect you from malware or ransomware

3. RAID5 is absolutely not sufficient
In case of failure (one disk died) and you insert a new one, the RAID undergoes the worst scenario: It rebuilds for hours and days, Experienced administrators know, that right in this moment it is very likely that another disk collapses under the load and caused heat. Therefore you should mirror a complete NAS or at least go for a RAID6 or ZFS+2.
Never run a RAID5 without a backup! A RAID is no replacement for a backup.
Jerry Yesavage avatar
Jerry Yesavage avatar
Sean Mc:
Online storage only for me. I have a 1.5 gigabit internet connection. Amazon has far better redundancy than I ever will.

Surprised nobody mentions AWS "glacier" storage.  What we use at work for research data. 

What are your costs running...???

Here is what MS CoPilot sez... more pricey than I thought (never see bills at work) :

Amazon S3 Glacier offers a cost-effective solution for long-term storage of data. Let’s explore the pricing for storing 20TB of data per month:
  1. Standard Storage (S3 Glacier):
    • The standard storage cost for S3 Glacier is approximately $0.005 per GB per month.
    • To calculate the monthly cost for 20TB (20,000 GB):
      • 20,000 GB × $0.005/GB = $100 per month.

  2. Free Tier Allowance:
    • As part of the AWS Free Usage Tier, you can retrieve up to 10GB of your S3 Glacier data per month for free.
    • The free tier allowance applies to Standard retrievals.

  3. Data Transfer Pricing:
    • Data transfer in to Amazon S3 Glacier is always free.
    • Data transfer out from Amazon S3 Glacier to other AWS regions costs $0.02 per GB.

Remember that these prices are approximate and may vary based on your specific AWS region and usage patterns.  If you need more accurate pricing or have specific requirements, consider requesting a pricing quote directly from AWS.
Ryder Cobean avatar
Ryder Cobean avatar
Jerry Yesavage:
Sean Mc:
Online storage only for me. I have a 1.5 gigabit internet connection. Amazon has far better redundancy than I ever will.

Surprised nobody mentions AWS "glacier" storage.  What we use at work for research data. 

What are your costs running...???

Here is what MS CoPilot sez... more pricey than I thought (never see bills at work) :

Amazon S3 Glacier offers a cost-effective solution for long-term storage of data. Let’s explore the pricing for storing 20TB of data per month:
  1. Standard Storage (S3 Glacier):
    • The standard storage cost for S3 Glacier is approximately $0.005 per GB per month.
    • To calculate the monthly cost for 20TB (20,000 GB):
      • 20,000 GB × $0.005/GB = $100 per month.

  2. Free Tier Allowance:
    • As part of the AWS Free Usage Tier, you can retrieve up to 10GB of your S3 Glacier data per month for free.
    • The free tier allowance applies to Standard retrievals.

  3. Data Transfer Pricing:
    • Data transfer in to Amazon S3 Glacier is always free.
    • Data transfer out from Amazon S3 Glacier to other AWS regions costs $0.02 per GB.

Remember that these prices are approximate and may vary based on your specific AWS region and usage patterns.  If you need more accurate pricing or have specific requirements, consider requesting a pricing quote directly from AWS.

This is what I do! I have a bucket set objects to Intelligent Tiering so as they age out of me actively working with them, they go to Glacier. I do my stacking in AWS as well with an EC2 instance that I spin up temporarily from an AMI to do the job, then push the masters back to S3 and kill it. Works great unless I forget to turn off the VM and run up a bill for a day! Need to implement a maximum uptime killswitch on it.
Jerry Yesavage avatar
Jerry Yesavage avatar
Unfortunately I am responsible for some 80TB of research data at a major University research center and we set up out backups when AWS first came out way back when...  AWS was very helpful getting this right the first time and have not looked back.  We have three servers here that have the primary copies of the data.  Two windows servers that are mirrored and then the AWS is the third leg of our backup... off site.  We have a third Linux serer for data processing.  My astroimaging total is only? 12TB and I have 20TB storage mirrored on my latest W11 machine.  Unfortunately I now have an ASI 6200 with huge images.  Decided long ago not to keep the primary images.  Anyway, will probably retire from astrophotography before I fill up the servers.... [I hope]