ASI2600 File Size Management

I use an expandable NAS to store the data. At 50MB per image, it adds up quick. For my workstation, I have a 1TB SSD for OS and apps, and a 1TB SSD for images that I'm currently processing. Once I'm done processing, I move the data to the NAS.

I operate remote, using ASI6200 cameras, subframe files are 120 meg. For each camera I have 1TB local storage on the respective control computer, and a 10TB NAS at the remote site for backup. For processing I have a 14Tb working drive and a larger rack mounted NAS-mirror drive for archival storage.

I use Dropbox to store everything. You need to buy space there, but everything is secure, retrievable at any time, and not that expensive.
best,

Vivian

Vivian Budnik:
I use Dropbox to store everything. You need to buy space there, but everything is secure, retrievable at any time, and not that expensive.
best,

Vivian

The problem I have with that is the cost. I could put together a NAS with about 10TB of storage for about $525. From what I can see dropbox costs $16.58/month for only 3TB of storage. So it would only take 2.5ish years to break even.

It's the same reason I bought Pixinsight instead of going with Photoshop. $260 once vs $21 for the rest of time. Only 12 months for that one.

If you are using NINA consider enabled .xisf file compression. It will drastically decrease data usage.

Stuart Taylor:

If you are using NINA consider enabled .xisf file compression. It will drastically decrease data usage.

This sounds interesting. I use NINA (2.0) but I can't see this feature. Can you point me to it?
Is the compression lossy at all? There is presumably some downside?

Under Options > Imaging > "Save image as" set to xisf. Then set compression to LZ4 and SHA-1, and enable "Byte Shuffling".

The compression is lossless. The downside is that you can only open the files in PixInsight (which may not be negative depending on your workflow) and they take slightly more CPU horsepower to decompress.

<deleted>  (Sorry, I was mentioning .xisf before noting it was already discussed)

Stuart Taylor:

If you are using NINA consider enabled .xisf file compression. It will drastically decrease data usage.

This sounds interesting. I use NINA (2.0) but I can't see this feature. Can you point me to it?
Is the compression lossy at all? There is presumably some downside?

Under Options > Imaging > "Save image as" set to xisf. Then set compression to LZ4 and SHA-1, and enable "Byte Shuffling".

The compression is lossless. The downside is that you can only open the files in PixInsight (which may not be negative depending on your workflow) and they take slightly more CPU horsepower to decompress.

great! Thanks. I only use Pixinsight so that sounds good!

I operate remote, using ASI6200 cameras, subframe files are 120 meg. For each camera I have 1TB local storage on the respective control computer, and a 10TB NAS at the remote site for backup. For processing I have a 14Tb working drive and a larger rack mounted NAS-mirror drive for archival storage.

Like Dave, I use a NAS system to back up the 120MB files from my remote system in Chile.  It is set up to be fully automatic so that the data just appears on my NAS drive as it is taken.  That one scope typically generate 5-8 GB/day and with nearly 300 clear nights/yr that requires a capacity of 1.5 - 2.5 TB/yr.  I'm about to install a second scope which will double the storage requirements.  NAS systems are FAR less expensive than using something like Google Drive or Drop Box.  My system currently has 4x12 TB drives configured as SHR + RAID5 to give 36 TB of storage.  I have an additional, identical system offsite for backup.  I just use these NAS systems only for image data so they should last for roughly 18 years with one scope and another maybe 10 years with two scopes.  The system is expandable if I need more space.  Here's the system that I use:  https://www.synology.com/en-us/products/DS923+.  With four Seagate Ironwolf 12 TB drives this system cost around $1500.  You'll also want to install a good quality UPS and in my case, I had to upgrad my internet service to increase the upload speed to make it easier to download data remotely.  I may eventually upgrade my home internet to fiber to get symmetric data speeds (same upload and download speeds) but that's in the future.   It's not super cheap but it's a one-time investment that lasts for a decade.  If you look at what it would cost to save that much data on Google or Drop Box over that period of time, its 

One other thing about both Google Drive and Drop Box is that both serves have dropped their backup services and replaced them with "Sync" service.  Sync service is not suitable as pure data backup service since it keeps the target drive synchronized with the source drive.  That means that if you erase data on the observatory drive, it gets erased on the synchronized drive--and you lose it all!  That's bad.  NAS systems can be configured the same way -or- they can be configured as a true backup system, which is what you want!  The other problem with both Google Drive and Drop Box is that it's hard to down load a folder with all of the ~6 GB of data taken in a single night.  There are ways to do it but it's common to have the download stop and need to be restarted.  It is NOT a smooth process.


-John

Linwood Ferguson:
Incidentally, one space management technique I don't see mentioned:  get where you can shoot longer exposures.

200 x 60s exposures vs 100 x 120s vs 50 x 240s may not be perfectly equivalent, but can be similar (depending on target and sky).  If you can keep the stars round at longer exposures, this can save quite a lot of space.

Agreed.  This is one strategy but the other that I've suggested to SGP is to allow sub-frame imaging.  This is easily done in MaximDL and a few other programs but it is notably lacking in SGP, which I use for my 20" in Chile.  I often work on small targets that don't require the full field and I could save on both storage space and bandwidth requirement if I could simply specify a cropped region of the sensor that I want to work with.  That capability would allow lucky imaging on very small targets even with a remote system.  It will be interesting to see how long it takes for the guy at SGP to get around to it.

John

John Hayes:

Linwood Ferguson:
Incidentally, one space management technique I don't see mentioned:  get where you can shoot longer exposures.

200 x 60s exposures vs 100 x 120s vs 50 x 240s may not be perfectly equivalent, but can be similar (depending on target and sky).  If you can keep the stars round at longer exposures, this can save quite a lot of space.

Agreed.  This is one strategy but the other that I've suggested to SGP is to allow sub-frame imaging.  This is easily done in MaximDL and a few other programs but it is notably lacking in SGP, which I use for my 20" in Chile.  I often work on small targets that don't require the full field and I could save on both storage space and bandwidth requirement if I could simply specify a cropped region of the sensor that I want to work with.  That capability would allow lucky imaging on very small targets even with a remote system.  It will be interesting to see how long it takes for the guy at SGP to get around to it.

John

NINA has added some support for that, though another alternative with programs like Pixinsight (with good batch processing) is you can do a batch crop on all your images after the fact.  That doesn't save transfer speed or space on the imaging computer, but for long term storage it does help.

A downside of subframe or crops is if done pre-calibration you need similarly sized flats and darks and such.

I work on a local 4TB SSD. Then, once elaborated, I move all on external 14TB HDD. I keep masters and final image on the internal SSD.
The 14TB HDD has multiples bakup on a NAS. It has slower connection but with more space (about 36TB).