Dear all, I spent several months renewing my setup. Today, it works well. I still have to optimize my calculation times. I use PixInsight on my NoteBook. But it's slow… Even with an i7, 16GB of RAM and 1TB of SSD. Buying a really muscular workstation doesn't seem to me to be relevant. It's very expensive and quickly obsolete for hobby use. I would rather rent CPU time on external servers. Do you know of any providers that rent? Have any of you found an interesting solution? Thanks in advance for your comments. Clear skyChristian
Buy a powerful workstation or rent remote CPU time on huge servers?
Engaging
Hey there.
I recently purchased an AMD threadripper 3990x as well as a few other upgrades like SATA to NVME drives, etc..
I wrote about my observations and test results here https://www.cloudynights.com/topic/757851-amd-threadripper-3990x-reviewthoughts/#entry10911246
Based on what I observed and what I see from the general community you can most likely get really good snappy results using much cheaper gear than what I spent. Setting up ramdisks or using Linux will result in even more performance with PixInsight specifically.
I'd also recommend a decent GPU, since upgrading to an NVIDIA Titan X my Starnet++ jobs are insanely fast for even stride 32.
Overall I'd say you should invest $600-800 in a decent "gaming" type computer with fast compute and a decent GPU, and setup ramdisks. And you will likely get very good results, as well as have a PC you own and can use for other things, vs renting space.
I recently purchased an AMD threadripper 3990x as well as a few other upgrades like SATA to NVME drives, etc..
I wrote about my observations and test results here https://www.cloudynights.com/topic/757851-amd-threadripper-3990x-reviewthoughts/#entry10911246
Based on what I observed and what I see from the general community you can most likely get really good snappy results using much cheaper gear than what I spent. Setting up ramdisks or using Linux will result in even more performance with PixInsight specifically.
I'd also recommend a decent GPU, since upgrading to an NVIDIA Titan X my Starnet++ jobs are insanely fast for even stride 32.
Overall I'd say you should invest $600-800 in a decent "gaming" type computer with fast compute and a decent GPU, and setup ramdisks. And you will likely get very good results, as well as have a PC you own and can use for other things, vs renting space.
Helpful
Hey there.
I recently purchased an AMD threadripper 3990x as well as a few other upgrades like SATA to NVME drives, etc..
I wrote about my observations and test results here https://www.cloudynights.com/topic/757851-amd-threadripper-3990x-reviewthoughts/#entry10911246
Based on what I observed and what I see from the general community you can most likely get really good snappy results using much cheaper gear than what I spent. Setting up ramdisks or using Linux will result in even more performance with PixInsight specifically.
I'd also recommend a decent GPU, since upgrading to an NVIDIA Titan X my Starnet++ jobs are insanely fast for even stride 32.
Overall I'd say you should invest $600-800 in a decent "gaming" type computer with fast compute and a decent GPU, and setup ramdisks. And you will likely get very good results, as well as have a PC you own and can use for other things, vs renting space.
I essentially did the same thing and it has worked out perfectly. Get an AMD threadripper gaming-type computer and you will have everything at your immediate disposal. I use mine for both astrophotography and my consulting business.
Sorry I cannot help. I was not aware that you could rent processing remotely. But I am very interested in hearing about it.
One concern that I have on using such a service is purely theoretical. The part that takes the most time is the preprocessing, calibration, registration and stacking. If this were the same for you, then consider that even if you were able to access a fast CPU elsewhere, you would still have to get all the data to that CPU. And considering the file size vs the bandwidth, you may be waiting a lot longer for that activity because of file transfer speeds. Remember, the CPU cannot reach back to your laptop do the job locally and then retreat to its home! You probably could calculate what the file transfer times you would endure very accurately based on your typical upload speeds. Recall, most internet service upload speeds (which is what you will require) are but a fraction of the download speeds you pay for. If you need to upgrade your service, that will be a premium in cost.
Even if you are willing to endure long upload times, say during sleep time, recall that if you are willing to use the current, and pretty flexible weighted batch preprocessing routines, they too can work during your off time. Again, very few other features are that time intensive, given a competent local computer.
CS
Alan
One concern that I have on using such a service is purely theoretical. The part that takes the most time is the preprocessing, calibration, registration and stacking. If this were the same for you, then consider that even if you were able to access a fast CPU elsewhere, you would still have to get all the data to that CPU. And considering the file size vs the bandwidth, you may be waiting a lot longer for that activity because of file transfer speeds. Remember, the CPU cannot reach back to your laptop do the job locally and then retreat to its home! You probably could calculate what the file transfer times you would endure very accurately based on your typical upload speeds. Recall, most internet service upload speeds (which is what you will require) are but a fraction of the download speeds you pay for. If you need to upgrade your service, that will be a premium in cost.
Even if you are willing to endure long upload times, say during sleep time, recall that if you are willing to use the current, and pretty flexible weighted batch preprocessing routines, they too can work during your off time. Again, very few other features are that time intensive, given a competent local computer.
CS
Alan
Helpful Insightful Respectful Engaging Supportive
Did you think of an Apple with the new M1 processor? You could buy a MacBook Air, a MacBook Pro, a Mac Mini or an iMac. These computers are not expensive but are fast. Very fast!
And they run PixInsight, what is important for you.
Just read or look some reviews. It's worth a deep thought.
And they run PixInsight, what is important for you.
Just read or look some reviews. It's worth a deep thought.
Depending on how much data you are wanting to process don't forget about the time it would cost to upload and download all the data. Recently I have been doing project with at least 150GB data and I have fast broadband connection but uploading all that data would still cost more time than running it on my own desktop. The more projects I do and data I gather the bigger a challenge the remote option would become.
Concise
I would also suggest you run Linux, Linux Mint would be an easy transition from Windows. I run Linux Mint with AMD 8core 32 GB Ram 1TB Nvme SSD for OS and data. it is a lot faster than my windows box. Not sure about the GPU, I didn’t think Pixinsight used the GPU, I know you can set Starnet to use it but that’s all I think.
I would also suggest you run Linux, Linux Mint would be an easy transition from Windows. I run Linux Mint with AMD 8core 32 GB Ram 1TB Nvme SSD for OS and data. it is a lot faster than my windows box. Not sure about the GPU, I didn’t think Pixinsight used the GPU, I know you can set Starnet to use it but that’s all I think.
Yeah Starnet uses the GPU, nothing else in pixinsight does AFAIK.
Problems about data transfer speeds aside, I actually checked that. Were I to rent the CPU power I have at home for the time I use it, I would spend the price of the computer every 6 months for AWS instances (including data transfer and storage of interim files, assuming everything is deleted once it is not needed). This could be reduced somewhat by doing the data transfer with a low-powered instance for even morr hassle. Some other clouds are cheaper, but not significantly enough. For a private hobby the added cost is just too much, since you pay a lot for added flexibility, the possibility to instantly scale and reduced/ease of administration that are only benefits in a corporate context.
True CPU speed increases for desktop/workstation use are still far between, even though pace has picked up a bit recently with the advances by AMD and Apple. So obsolescence is not that much of a problem. At the moment I‘ld wait a bit and see what Apple comes up with, the M1 is very promising.
My experience has also been that notebooks (like my Macbook Pro) are really limited by thermals: The design is not cut out for lengthy stretches of 100% CPU use. Heat will build up and the CPU is throttled. This is much better with a desktop/workstation.
You did not specifiy how long is „too long“ but I‘ve run calibrations that even my Threadripper setup (3960X) would need hours to complete. Sometimes it is limited by I/O, sometimes by a lack of parallelism in PI preprocessing. A good analysis of your setup and tuning (especially increasing the amount of I/O threads and adding a ramdisk for caching) might speed up your notebook significantly.
True CPU speed increases for desktop/workstation use are still far between, even though pace has picked up a bit recently with the advances by AMD and Apple. So obsolescence is not that much of a problem. At the moment I‘ld wait a bit and see what Apple comes up with, the M1 is very promising.
My experience has also been that notebooks (like my Macbook Pro) are really limited by thermals: The design is not cut out for lengthy stretches of 100% CPU use. Heat will build up and the CPU is throttled. This is much better with a desktop/workstation.
You did not specifiy how long is „too long“ but I‘ve run calibrations that even my Threadripper setup (3960X) would need hours to complete. Sometimes it is limited by I/O, sometimes by a lack of parallelism in PI preprocessing. A good analysis of your setup and tuning (especially increasing the amount of I/O threads and adding a ramdisk for caching) might speed up your notebook significantly.
Helpful
Yeah Starnet uses the GPU, nothing else in pixinsight does AFAIK.
Out of the box, PixInsight does not use the GPU for Starnet. There is an unofficial workaround for accelerating Starnet on (some) NVidia GPUs.
Also PixInsight hasn't been optimized for M1 processors yet, but some M1 Mac users report good results via the Rosetta dynamic binary translator (offered by Apple as a compatibility layer).
Torben van Hees:
My experience has also been that notebooks (like my Macbook Pro) are really limited by thermals: The design is not cut out for lengthy stretches of 100% CPU use. Heat will build up and the CPU is throttled. This is much better with a desktop/workstation.
This is true. And the extra benefit of having a desktop is ergonomics. You don't ruin your posture (as I do myself right now LOL!) by hunching in front of a tiny (and occasionally hot) keyboard and monitor.
Helpful Engaging
Die Launische Diva:
This is true. And the extra benefit of having a desktop is ergonomics. You don't ruin your posture (as I do myself right now LOL!) by hunching in front of a tiny (and occasionally hot) keyboard and monitor.
My laptop sits in a stand connected to a few docking stations and some external storage. It is used with a full sized keyboard and a 30“ screen. Through the docking stations, about 25 USB peripherals are connected (mostly MIDI decices). The workstation is in a rack in the basement where it can blow its loud fans all day and night without disturbing me. I use it via VNC.
Well written
Voirol Christian:
Dear all, I spent several months renewing my setup. Today, it works well. I still have to optimize my calculation times. I use PixInsight on my NoteBook. But it's slow... Even with an i7, 16GB of RAM and 1TB of SSD. Buying a really muscular workstation doesn't seem to me to be relevant. It's very expensive and quickly obsolete for hobby use. I would rather rent CPU time on external servers. Do you know of any providers that rent? Have any of you found an interesting solution? Thanks in advance for your comments. Clear skyChristian
First, how big are your image files and how many do you process for each image? I use a MacBook Pro with an I9 processor and 64MB of RAM (with 2TB SSD.) Using PI, it chews right through hundreds of 32 MB images from my FLI-ML16803 without much trouble. I now have a QHY600PH-M on my 20” and that camera produces 160 MB images. That definitely slows things down but it’s still manageable.
John
Well written Helpful Concise Engaging Supportive
Dear all, Thank you for your comments, remarks and suggestions. I made several tests and the results are presented in French and in English in the attached file. So I won't repeat what is there. Concerning the size of the files: so far, my spring target (which is not completed... but it's coming), M101, is made of more than 800 files of 22MB each (bias, dark, flat, light, that is to say about 15GB of data). Using the VM described in the file, everything runs better. To date, I'm stuck on Azure because I don't know how to create a shared virtual disk that I can access from my notebook and from the virtual machine. It's a simple technical issue, but the basic subscription doesn't provide technical support and I haven't found a resource on UpWork that can help me yet. But if I can find it, then I can transfer my 16GB slowly from my notebook, then run the processing on the virtual machine. I did several hours of testing this weekend (about 4-5 hours. Each hour costs about 8 euros). Total: 49$ for CPU (64-cores and 512GB RAM) and enough SSD. It works very well. Only problem: the possibility to see the changes live (e.g. Histogram Transform or Cosmetic Correction) does not work in remote. So you have to test your settings locally on a few files. The idea is to be able to use a very powerful machine just when needed. Here, there is no subscription. I connect and I am charged per use. Renting a shared permanent disk doesn't seem to be very expensive (but you have to be able to configure it... which I can't do at the moment). In any case, the 100GB of the disk containing the OS is permanent and free. Let's say I process my data one week-end on two for 5-6 hours, that makes a budget of about 2x50 euros per month, or CHF 1200 per year. For the AMD Ryzen Threadripper 3990X (4.3 GHz Max.), in Switzerland, it is more than CHF 5'000.-. And as a colleague said, I work from my sofa with a tablet on my lap. So, the desktop + screen, I don't know where to put it... And I don't have a desk. So at this point, renting seems like a feasible option. A colleague on another site suggested a French provider OVH. I'm waiting for their return so I can test it. I don't know everything yet, but I'll get back to you as soon as I know more.
Clear sky
Christian
210703 - Evaluation finale - Virtual Machines WorkStations.pdf
Clear sky
Christian
210703 - Evaluation finale - Virtual Machines WorkStations.pdf
Helpful
Not meaning to highjack this thread, I would like to mention that PixInsight is not the only and not the most well-written software for astrophotography speedwise. Other software might be faster. I use Siril. Stacking 60 images of the ASI 6200MM with 9500x6200 pixel is ony a matter of a minute. This is because Siril is written for speed from the start. It makes much better use of the multiprocessing architecture of your PC and also the memory. Since Siril is free, it might be worth a try.
Helpful Concise
Voirol Christian:
I did several hours of testing this weekend (about 4-5 hours. Each hour costs about 8 euros). Total: 49$ for CPU (64-cores and 512GB RAM) and enough SSD. It works very well. Only problem: the possibility to see the changes live (e.g. Histogram Transform or Cosmetic Correction) does not work in remote. So you have to test your settings locally on a few files.
*Very interesting Voirol. Is the process tended too while in process? The setup seems powerful for processing, and this may work flawlessly, but even in the best of times, my computer will hang on a particular job for no reason and I have to restart, or at least pick up from where it died. How is such a situation handled if it were to occur? (And how would it be billed?) But for your situation, it seems ideal, assuming you can work out the data transfer issues.
Alan
One of the most important things when using pixinsight is having as many threads as possible. Pixinsight is a thread monster, so that should come first when searching for a build. You can go the way stobi went and get a threaddripper, but I suspect you dont need it to be that fast(but oh boy do I want one!). GPU doesnt matter, I have a cheap $100 GPU.
I will get you a list of my build shortly. Pixinsight benchmark for me is around 22 seconds. My laptop I used to use was 2.5 MINUTES.
Here is my build: https://pcpartpicker.com/list/w2K6mk
I would hold off though. I built my PC back in January for about $1000....Before that it could be done for under $1,000. I would not pay $1750 that it is suggesting it is.
I will get you a list of my build shortly. Pixinsight benchmark for me is around 22 seconds. My laptop I used to use was 2.5 MINUTES.
Here is my build: https://pcpartpicker.com/list/w2K6mk
I would hold off though. I built my PC back in January for about $1000....Before that it could be done for under $1,000. I would not pay $1750 that it is suggesting it is.
Helpful Engaging Supportive
Voirol Christian:
Dear all, I spent several months renewing my setup. Today, it works well. I still have to optimize my calculation times. I use PixInsight on my NoteBook. But it's slow... Even with an i7, 16GB of RAM and 1TB of SSD. Buying a really muscular workstation doesn't seem to me to be relevant. It's very expensive and quickly obsolete for hobby use. I would rather rent CPU time on external servers. Do you know of any providers that rent? Have any of you found an interesting solution? Thanks in advance for your comments. Clear skyChristian
You never mention if you are seeing paging between memory to the SSD and I suspect by the volume of data you are dealing with, you might see a significant boost with a memory upgrade if you machine will support it. Windows is a demand page OS, like all modern virtual memory OS's, and that means it will page to physical memory first, then to disk when that runs low. Memory is many orders of magnitude faster than any SSD so it could make a huge difference in your performance.
FWIW,
To answer the original question: I looked into this back before I build my new desktop in jan 2020 (I guess that's a spoiler). Since I work in cloud/IT, figured I'd look around and see what the options are.
To keep it simple:
The vast majority of "cloud" hardware is server hardware. Not that it can't be used for our purposes, but in general it favors reliability and consistency over raw performance. More importantly, while it is somewhat competitively priced compared to the total cost of ownership of an enterprise datacenter server with sysadmins that need to be paid, cooling, power redundancy, network availability, physical and network security, etc… It is not, by a long shot, price-competitive with todays cheap off-the-shelf consumer hardware or even with homelab DIY server setups if you are using it semi-frequently.
There's a couple of options out there that offer "cloud desktops" or "virtual workstations" but most of those are meant for desk workers running Excel and Outlook 90% of the day (it's a way to keep the data of client devices in a BYOD or work from home setup). Those aren't going to have the oomph you are looking for and again, you are paying an "enterprise" premium. These services also tend to either charge monthly (not hourly or daily) and/or have a minimum number of users. Should you wish to run Windows, licensing for RDP and CALs can also become a mess not worth your time.
With most cloud solutions there's also data storage and network bandwidth fees that apply on top of CPU time used. For at least the data storage part the bill keeps running even when not in use.
Long story short, a home desktop machine will kick the snot out of any laptop for extended high load (just thermodynamics, the heat needs to go somewhere) as well as having greater longevity if you upgrade piecemeal along the line. My previous machine (Haswell Core i7 4770K from 2013) still sees daily usage by the other members of the household (desktop and light gaming) so you can really get some mileage out of your hardware if you know what you are doing.
The only problem right now is that hardware prices (and esp GPUs) have gone crazy with all that is going on (supply issues, scalpers, cryptocoins), if you can postpone a bit until prices settle down again (there's already a downward trend) you won't regret it.
No need to go overboard with threadripper or somesuch, an AMD 5600X (5800X or 5900X if it's in your budget) will do if you pair it with a fast SSD and plenty of RAM. A 1000$ budget (under normal circumstances) will go a long while for a system that you can use at least 5 years (and then upgrade piecemeal if you want, replacing some components keeping others - my PSU is from 2008 ). Assuming you use it weekly, a cloud option should be less than 4$ / session to beat that.
To keep it simple:
The vast majority of "cloud" hardware is server hardware. Not that it can't be used for our purposes, but in general it favors reliability and consistency over raw performance. More importantly, while it is somewhat competitively priced compared to the total cost of ownership of an enterprise datacenter server with sysadmins that need to be paid, cooling, power redundancy, network availability, physical and network security, etc… It is not, by a long shot, price-competitive with todays cheap off-the-shelf consumer hardware or even with homelab DIY server setups if you are using it semi-frequently.
There's a couple of options out there that offer "cloud desktops" or "virtual workstations" but most of those are meant for desk workers running Excel and Outlook 90% of the day (it's a way to keep the data of client devices in a BYOD or work from home setup). Those aren't going to have the oomph you are looking for and again, you are paying an "enterprise" premium. These services also tend to either charge monthly (not hourly or daily) and/or have a minimum number of users. Should you wish to run Windows, licensing for RDP and CALs can also become a mess not worth your time.
With most cloud solutions there's also data storage and network bandwidth fees that apply on top of CPU time used. For at least the data storage part the bill keeps running even when not in use.
Long story short, a home desktop machine will kick the snot out of any laptop for extended high load (just thermodynamics, the heat needs to go somewhere) as well as having greater longevity if you upgrade piecemeal along the line. My previous machine (Haswell Core i7 4770K from 2013) still sees daily usage by the other members of the household (desktop and light gaming) so you can really get some mileage out of your hardware if you know what you are doing.
The only problem right now is that hardware prices (and esp GPUs) have gone crazy with all that is going on (supply issues, scalpers, cryptocoins), if you can postpone a bit until prices settle down again (there's already a downward trend) you won't regret it.
No need to go overboard with threadripper or somesuch, an AMD 5600X (5800X or 5900X if it's in your budget) will do if you pair it with a fast SSD and plenty of RAM. A 1000$ budget (under normal circumstances) will go a long while for a system that you can use at least 5 years (and then upgrade piecemeal if you want, replacing some components keeping others - my PSU is from 2008 ). Assuming you use it weekly, a cloud option should be less than 4$ / session to beat that.
Helpful
Voirol Christian:
Dear all, I spent several months renewing my setup. Today, it works well. I still have to optimize my calculation times. I use PixInsight on my NoteBook. But it's slow... Even with an i7, 16GB of RAM and 1TB of SSD. Buying a really muscular workstation doesn't seem to me to be relevant. It's very expensive and quickly obsolete for hobby use. I would rather rent CPU time on external servers. Do you know of any providers that rent? Have any of you found an interesting solution? Thanks in advance for your comments. Clear skyChristiaIn
I'm an experienced high performance systems engineer and would not recommend cloud-based solutions for several reasons. For one thing, software licensing is usually per-seat (i.e. per-cpu). I'm not sure about PixInsight, but commercial software solutions typically limit you to using the product on one or two multicore CPUs installed on one machine only with a non-transferrable license. On top of that, depending on where you're located with respect to the cloud provider, network latency could be a problem. If you were able to resolve these concerns, cloud providers typically charge by cpu cycle and memory throughput. So you have a multi-threaded app that uses 2-4 quad-core cpus, and you're hitting max throughput, costs could go up quickly. I'm usually working with scientists running fluid dynamics models, so I will admit I'm not that experienced with estimating costs for a home user executing image processing tasks. However, I will adamantly state that it would be much better to get an AMD-based gaming machine, or even a refurbished Dell Precision workstation with a quad core Xeon processor and at least 16 GB ram and a decent GPU like an Nvidia Quadro/Geforce, or an AMD FirePro/Radeon (avoid integrated graphics). Right now, you can get a very good Dell Precision workstation with 32 GB RAM and a quad-core Xeon for less than $1,000 USD on the Dell Refurbished website.
Helpful Respectful Engaging Supportive
David Goldstein:Voirol Christian:
Dear all, I spent several months renewing my setup. Today, it works well. I still have to optimize my calculation times. I use PixInsight on my NoteBook. But it's slow... Even with an i7, 16GB of RAM and 1TB of SSD. Buying a really muscular workstation doesn't seem to me to be relevant. It's very expensive and quickly obsolete for hobby use. I would rather rent CPU time on external servers. Do you know of any providers that rent? Have any of you found an interesting solution? Thanks in advance for your comments. Clear skyChristian
You never mention if you are seeing paging between memory to the SSD and I suspect by the volume of data you are dealing with, you might see a significant boost with a memory upgrade if you machine will support it. Windows is a demand page OS, like all modern virtual memory OS's, and that means it will page to physical memory first, then to disk when that runs low. Memory is many orders of magnitude faster than any SSD so it could make a huge difference in your performance.
FWIW,
Yes, it's true. I currently have 1TB of SSD and I could upgrade my RAM to 32GB or 64GB. But this is my professional machine... And the Service Info refuses even if I pay... So I have to leave it at that. Hence the question: do I buy or rent?
Götz Golla:
Not meaning to highjack this thread, I would like to mention that PixInsight is not the only and not the most well-written software for astrophotography speedwise. Other software might be faster. I use Siril. Stacking 60 images of the ASI 6200MM with 9500x6200 pixel is ony a matter of a minute. This is because Siril is written for speed from the start. It makes much better use of the multiprocessing architecture of your PC and also the memory. Since Siril is free, it might be worth a try.
Indeed, this may be the next step. But as it is, I'm on PI and I'd like to go around it. I already find it difficult to understand what each treatment does. But I must admit that I like this software. I see how to go further. I program in Python (after assembler, Fortran 66-77, basic, ADA, C++, etc. in my younger years)... I see that it is possible to program macros in Java... This is a perspective that interests me. And maybe Syril does that too...
Benny Colyn:
To answer the original question: I looked into this back before I build my new desktop in jan 2020 (I guess that's a spoiler). Since I work in cloud/IT, figured I'd look around and see what the options are.
To keep it simple:
The vast majority of "cloud" hardware is server hardware. Not that it can't be used for our purposes, but in general it favors reliability and consistency over raw performance. More importantly, while it is somewhat competitively priced compared to the total cost of ownership of an enterprise datacenter server with sysadmins that need to be paid, cooling, power redundancy, network availability, physical and network security, etc... It is not, by a long shot, price-competitive with todays cheap off-the-shelf consumer hardware or even with homelab DIY server setups if you are using it semi-frequently.
There's a couple of options out there that offer "cloud desktops" or "virtual workstations" but most of those are meant for desk workers running Excel and Outlook 90% of the day (it's a way to keep the data of client devices in a BYOD or work from home setup). Those aren't going to have the oomph you are looking for and again, you are paying an "enterprise" premium. These services also tend to either charge monthly (not hourly or daily) and/or have a minimum number of users. Should you wish to run Windows, licensing for RDP and CALs can also become a mess not worth your time.
With most cloud solutions there's also data storage and network bandwidth fees that apply on top of CPU time used. For at least the data storage part the bill keeps running even when not in use.
Long story short, a home desktop machine will kick the snot out of any laptop for extended high load (just thermodynamics, the heat needs to go somewhere) as well as having greater longevity if you upgrade piecemeal along the line. My previous machine (Haswell Core i7 4770K from 2013) still sees daily usage by the other members of the household (desktop and light gaming) so you can really get some mileage out of your hardware if you know what you are doing.
The only problem right now is that hardware prices (and esp GPUs) have gone crazy with all that is going on (supply issues, scalpers, cryptocoins), if you can postpone a bit until prices settle down again (there's already a downward trend) you won't regret it.
No need to go overboard with threadripper or somesuch, an AMD 5600X (5800X or 5900X if it's in your budget) will do if you pair it with a fast SSD and plenty of RAM. A 1000$ budget (under normal circumstances) will go a long while for a system that you can use at least 5 years (and then upgrade piecemeal if you want, replacing some components keeping others - my PSU is from 2008 ). Assuming you use it weekly, a cloud option should be less than 4$ / session to beat that.
Thank you for these comments which are in line with my questioning. I am still wondering about the 1000 euros machine you are talking about... From my part of the country, a machine like the one described is closer to 3000 euros than to 1000 euros (https://www.toppreise.ch/chercher?q=ryzen%209%205900x). And I still need a screen, a mouse, etc...
Basically, my questioning evolves and so does my practice. I've just spent 3 months on M101. I have collected a decent number of images. Each time I go out (2x/month or so... It rains all the time), I integrate my images once to see what it looks like. There are not that many, it takes an hour or two of cloud. Then, if I stay with this logic, the complete integration will take several hours of computing once per season. In any case, I'm on a roll, I'm going to complete the trial.
For the configuration that I can't finalize alone, I followed AZURE's advice and went on UpWork. I submitted a request and against all odds, I got plenty of responses. All from systems engineers who seemed to understand the details of the configuration. I hired someone from Mauritius and he is going to help me early this week. I will see if it works
Hello again,
In order to close the loop, here is a final summary of my trials with Azure virtual machines (VMs). First of all, I was not able to configure my machine by myself. If you don't pay 30 euros per month for the subscription with 24/7 technical support, it's complicated. I want to limit my fixed costs so that I only pay when I use the VM. But Microsoft suggests us to use UpWork to hire freelance technicians. So I did that and it worked very well.
Second, my current setup is a 64-core VM with 512GB of RAM and 100GB of permanent shared SDD. From my home PC, I quietly transfer my raw to the shared drive before doing the processing on the VM. My first use was processing in PixInsight (WBPP 2.1) my nearly 1000 files (Flats, Darks, Lights, etc. collected over 6 separate sessions) to create the image found here: https://astrob.in/uzh5nk/F/. The machine took about 30 minutes for the complete processing. Then I quietly retrieved my Masters from my shared drive.
The second use was an intensive use of Deconvolution in PixInsight. I didn't quite understand what the many options were for. However, in a little over 2 hours, I was able to do a lot of testing by increasing the iterations beyond 100, changing various parameters, etc. I use the Windows screenshot to document my tests.
Finally, the monthly rental cost of the 100GB of permanent SDD is CHF 0.78 (about 25 euros/month) + CHF 7.68 per VM hour. Since I only use it for very memory and CPU intensive jobs (the rest is done on my personal laptop), it remains financially very reasonable. That's it. So I'll give up buying a big machine and do my integrations and other complex jobs directly on the Azure VM.
Thank you for your collaboration in the reflection.
Clear sky.
Christian
In order to close the loop, here is a final summary of my trials with Azure virtual machines (VMs). First of all, I was not able to configure my machine by myself. If you don't pay 30 euros per month for the subscription with 24/7 technical support, it's complicated. I want to limit my fixed costs so that I only pay when I use the VM. But Microsoft suggests us to use UpWork to hire freelance technicians. So I did that and it worked very well.
Second, my current setup is a 64-core VM with 512GB of RAM and 100GB of permanent shared SDD. From my home PC, I quietly transfer my raw to the shared drive before doing the processing on the VM. My first use was processing in PixInsight (WBPP 2.1) my nearly 1000 files (Flats, Darks, Lights, etc. collected over 6 separate sessions) to create the image found here: https://astrob.in/uzh5nk/F/. The machine took about 30 minutes for the complete processing. Then I quietly retrieved my Masters from my shared drive.
The second use was an intensive use of Deconvolution in PixInsight. I didn't quite understand what the many options were for. However, in a little over 2 hours, I was able to do a lot of testing by increasing the iterations beyond 100, changing various parameters, etc. I use the Windows screenshot to document my tests.
Finally, the monthly rental cost of the 100GB of permanent SDD is CHF 0.78 (about 25 euros/month) + CHF 7.68 per VM hour. Since I only use it for very memory and CPU intensive jobs (the rest is done on my personal laptop), it remains financially very reasonable. That's it. So I'll give up buying a big machine and do my integrations and other complex jobs directly on the Azure VM.
Thank you for your collaboration in the reflection.
Clear sky.
Christian
Helpful Engaging
Voirol Christian:
From my home PC, I quietly transfer my raw to the shared drive before doing the processing on the VM. My first use was processing in PixInsight (WBPP 2.1) my nearly 1000 files
Very interesting! Speaking about disk space, 1000 raw files are about 50Gb + all the intermediate files generated by WBPP one can easily reach 500Gb+ during pre processing. Do you extend your drive space on the fly from 100Gb to 500Gb+ when you start tranfering your raw files and run WBPP process immidiately after?
If I understand correctly, the permanent 100Gb are those where you leave your Pixinsight installed in order not to have install it every time, right?
And last, if you launch Pixinisight benchmark tool, how much does it take to complete the process and what's the score?
Voirol Christian:
Hello again,
In order to close the loop, here is a final summary of my trials with Azure virtual machines (VMs). First of all, I was not able to configure my machine by myself. If you don't pay 30 euros per month for the subscription with 24/7 technical support, it's complicated. I want to limit my fixed costs so that I only pay when I use the VM. But Microsoft suggests us to use UpWork to hire freelance technicians. So I did that and it worked very well.
Second, my current setup is a 64-core VM with 512GB of RAM and 100GB of permanent shared SDD. From my home PC, I quietly transfer my raw to the shared drive before doing the processing on the VM. My first use was processing in PixInsight (WBPP 2.1) my nearly 1000 files (Flats, Darks, Lights, etc. collected over 6 separate sessions) to create the image found here: https://astrob.in/uzh5nk/F/. The machine took about 30 minutes for the complete processing. Then I quietly retrieved my Masters from my shared drive.
The second use was an intensive use of Deconvolution in PixInsight. I didn't quite understand what the many options were for. However, in a little over 2 hours, I was able to do a lot of testing by increasing the iterations beyond 100, changing various parameters, etc. I use the Windows screenshot to document my tests.
Finally, the monthly rental cost of the 100GB of permanent SDD is CHF 0.78 (about 25 euros/month) + CHF 7.68 per VM hour. Since I only use it for very memory and CPU intensive jobs (the rest is done on my personal laptop), it remains financially very reasonable. That's it. So I'll give up buying a big machine and do my integrations and other complex jobs directly on the Azure VM.
Thank you for your collaboration in the reflection.
Clear sky.
Christian
I'd say if you had trouble on Azure, AWS is far far easier to configure and run workloads on than Azure. This is actually my day job (Cloud Architect). AWS has lightsail, workspaces, etc.. workspaces might be a good option, basically a virtual desktop where you specify the hardware, connect via the VDI workspaces app on mac or windows, and install PixInsight, etc.. You can stop/start the workspace as you need. And it's much much easier than anything else.
If you're linux savvy of course, linux is always the better option for this kind of stuff. And similar in AWS running EC2 instances, or Lightsail VM's, is easier than Azure in my experience.
Helpful
BlurXTerminator
I am starting to use BlurXTerminator and I am impressed. However I built my workstation 8 years ago and it takes 10 minutes to process BXT. It would be useful to know how long it takes other users and what in particular helps speed things up. I have ...
BlurXTerminator processing slow on older hardware; author seeks optimization solutions.
Budget Replacement GPU for Xterminator Processes
Hi Everyone, I have a three year old pc that I put together on a budget. Its current configuration is: AMD Ryzen 7 2700X cpu, Gigabyte B450M DS3H main board, 64Gb 3200Mhz DDR4 ram, 2 Gb gen 3 nvme m.2 , and a AMD FX-570 (4Gb) gpu. It does mostly what...
GPU upgrade discussion relevant to author's PixInsight performance optimization needs.
Laptop for Astrophotography Capture - Request for Advice
Hi all, When I started out with astrophotgraphy, I went with what looked like the prevailing guidance - that image capture doesn't need much computing power, and pretty much any computer will do. I bought a quite low-powered Windows laptop, think...
Laptop computing for astrophotography directly relates to author's notebook performance concerns.