PixInsight Processing Speed

I've been using DSS and Photoshop to process my images for the last few years and decided to give PI a try this week. I'm impressed with it and the wealth of resources out there to support the tool. The question I have is about the speed of the calibration steps.

I have a 16 GB RAM Surface which, using DSS, on a typical batch of 60 or so subframes from my ASI 6200 I'd expect to have them calibrated, cosmetically corrected, aligned and stacked within around 30 mins - all be it with quite a bit of manual intervention but still not an excessive amount of time. Testing PI yesterday I really liked the WBPP set up but when I loaded the files in and set it running even after 10 hours it was not done. I broke the package down into smaller batches say 12 files at a time and even then that was taking well over an hour to run.

I understand PI is more sophisticated and does bigger 'sums' etc but this duration is prohibitive - what I'm wondering is have I missed something or are you simply going to say 16 GB Ram how old school is that ;0).

I'd really like to shift to PI but the thought of dropping another $1500k minimum on a new laptop does not fill me with joy!

Many thanks

John

Yes, that's the reality, with full frame you need a beefy computer for the full-fledged PI stacking process. I am handling only APS-C and have 32GB with a 12-thread processor – I plan to upgrade as soon as the regular Intel 13-series chips become available next year.

Rafal, Thanks for the quick reply. I've compared the PI and DSS results, all be it just cosmetically not mathematically, and I'm not sure it makes that much of a difference. Of course PI is a much more integrated experience but given my 16 GB machine handles the DSS/PS workflow just fine and in real time even for the 6200 MM I'm hesitant to make the shift. I was hoping I done something dumb that would speed things up!

Thanks again

John

That does seem like a long time for stacking the subs. I have a Lenovo IdeaPad 3 with i7 core, 20GB ram on Windows 10. I'm using an ASI 2600mm with the APS-C sensor for acquisition. I typically stack around 24-28 subs in each filter in DSS (no calibration frames) and it approximates stacking time to be 2 hours but only takes about 4 minutes to register and stack. I also use PI with speedy results. The longest process in my PI workflow for my computer is using StarNet2 which usually takes around 4-5 minutes.

-Eddie

Within that script there are 7 processes being executed. (full disclosure, I don't use WBPP).

Image Calibration can take a while as it computes pedestals, analyzes Noise and SNR. Memory and threads help.
Cosmetic correction serially goes through the subs and doesn't take very long. Once you've loaded the files and chosen a reference frame to preview and applied your settings.
Blink - Have a look through your subs daily to ensure your calibration frames are working as desired and eliminate any subs that are just horrible.
Debayer - isn't very intensive, but can take a little while if you're separating the channels/
Subframe Selection - depends upon how many subs you're analyzing, but I've found I probably take more time looking at the information than it took to run the analysis. Outputting the subs to the approved folder doesn't take very long. Remember to note the best half dozen frames to blink so you can select your reference frame for Alignment & L_Norm.
Star Alignment - Memory and threads help this run faster, but honestly it's not too bad.
Local Normalization - This process takes a while. Especially if you have 300-500 frames, which is normal.
Image Integration - Brings all of the prior processing together. This can take a while too. Just depends upon the number of subs in your project. Again 300-500 is pretty normal.

If all of this is thrown into a script it will take quite a while - again depending upon the number of subs in your project. Memory and threads seem to be the key to speeding things up. A laptop might not be the best tool to use.

Cheers!

Thanks for the replies and understood on all the points made. I guess I'm still thinking why would I bother with PI for calibration etc when a laptop works just fine to do very similar things in DSS/PS in a fraction of the time. Also it appears I can bring all my calibrated files into PI from DSS and run DBE and PCC relatively quickly (both processes are better than my current work flow) but I would love to use the full power of PI for calibration as well. Hence my hope I've missed something in the set up. Clearly next time I'm out for a new computer I will go down the 64 GB route etc, but my current machine is only 9 months old so I'm guessing I'll stick with what I have for a while longer.

Thanks again

John

The default WBPP settings result in a computationally demanding preprocessing pipeline which might not be necessary for your data set. Try turning off local normalization and use a less demanding rejection algorithm like winsorized sigma clipping when performing integration. Finally, I would not recommend bringing calibrated files from other apps to PI.

I use this very laptop (a MacBook Air 2015 vintage with 8GB RAM and 250 GB SSD) to do the full calibration, registering, normalization and stack up to 60 21MPx light frames and it doesn't take that long (maybe 1/1.5 hour from start to finish). I'd never use WBPP myself and I have all the accessory files (master dark and master flat) already sorted. So, I suspect you should be able to achieve similar results with a much younger and faster machine with a larger memory.

The power of PI integration becomes apparent when you bring in material from many nights, with different illumination, perhaps different rotation and framing, or done with 2 different scopes/focal lengths. That's where all these intensive algorithms really show their worth.

One of the most powerful and game-changing aspects of the PI calibration is the Normalise Scale Gradient process - I don't believe there is an equivalent in DSS or APP, etc. With a reasonable PC and enough RAM the calibration flies through very quickly, with debayering being one of the longer processes, taking about 5s per image on my PC. Normalise Scale Gradient does take time though, as it only uses one core; this takes typically 15-20s per image. I'm using an AMD 5950X processor with 64GB of RAM for an APS-C sized OSC camera. I could probably get by with 32G of RAM, but it would run much slower as it would be swapping to virtual memory quite a bit for some operations.

John, I think you are following the right path by exploring aspects of Pixinsight. A more robust computer would be advantageous most definitely given your file sizes with your 6200.

@Monty Chandler has described the functions present in WBPP quite well and @Die Launische Diva is correct in that Local Normalization is the most time consuming and computer demanding function in WBPP.

There is a "Select" button on the right side of the WBPP page under the label "Presets" that gives you basically a fast, medium and slow choice of speed, and note that the default for WBPP is the slowest that includes local normalization which takes up the most computer power and time.
The fastest processing leaves local normalization out. I typically use the slower speeds when I am running tests to assess the quality of my stacked subs, session to session and I apply the full "Maximum Quality" process for final integrations.

You should try these other presets in Weighted Batch Preprocessing to see how they work for you with you current computer and continue expanding further into Pixinsight. As a side note the new default, PSF weight in Sub Frame selector under Imageinspection, is excellent.

CS,
scott

Turn off local normalisation and use Winsorized sigma clipping when first using WBPP. Also, you are probably RAM limited.

Again thanks all just for a bit more clarity if I look at the progress box in WBPP I believe the green highlight shows where I’m at in the flow? If so my last run after multiple hours hadn’t even progressed beyond the initial calibration never mind the more complex functions. Now perhaps they also incur a bunch of pre setup etc which might explain that.

Anyway bottom line I just can’t believe PI is 10 + times slower than DSS especially when I see Andreas post and he’s only got 8GB! It must be something I’m doing or running n the background. I’ve tried breaking into sets by frames and by processes and it’s still unusably slow compared to DSS.

I will keep working on it and let you know. Thanks again.

john

I'd venture to suggest you ditch WBPP for a second and do it the time-honored way, that is step-by-step. This way we'll see where the bottleneck is. Starting with the calibration process (I assume you have a master dark and a master flat ready at hand, if not create them). Here is the list:

1. ImageCalibration
2. CosmeticCorrection
3. Debayer (if OSC)
4. SubframeSelector (skip this step for now)
5. StarRegistration
6. LocalNormalization (skip this step for now)
7. ImageIntegration
8. DrizzleIntegration (only for OSC, with no oversampling, but skip this step for now)

Tell us how you're faring with each of them (times given in the Process Console)

John Noble:
but when I loaded the files in and set it running even after 10 hours it was not done.

This doesn't sound right at all ... I've never experienced integration times above one hour for that many frames. Maybe you could screen capture the progress window that shows hows much time is spent at each stage and share that with us. I run on an old core i7 8700 with 32G RAM and PI is really quite fast.

Some other thoughts: are you binning at all or are you at the full resolution? Are you using drizzle?

I do use WBPP occasionally but find for me APP does as good or better job for my data. Again this is just my opinion and I’m using an APS-C OSC camera, not the full frame that your using. I definitely agree your light on 16gigs of ram. I had that with my old i5 and with my newer computer I’m running 96gigs. Something you may want to do down the road is do a benchmark with your computer and add “swap file” folders in PI. That can help with your processing times as well. Here is a video that talks about that and how thats done. There are others out there as well. https://www.youtube.com/watch?v=cJ9oAA_9HCU

While PI can do everything that APP does, I feel that there are other powerful processes that you can take advantage of and like @andrea tasselli suggested, maybe back off of WBPP for a bit and learn/enjoy many of the other tools/processes PI has to offer. Once you get comfortable with it then maybe go back and work with it.

Also I would highly recommend watching Adam Blocks YouTube video’s on WBPP as well! He will help you understand more of what many here are explaining to you about the processing within WBPP. Here is a link to his YouTube channel. https://www.youtube.com/c/AdamBlock He has several on WBPP so start with the 1st one and follow the progression of WBPP and his updating of it.

Dale

Andy Wray:
John Noble:
but when I loaded the files in and set it running even after 10 hours it was not done.

This doesn't sound right at all ... I've never experienced integration times above one hour for that many frames. Maybe you could screen capture the progress window that shows hows much time is spent at each stage and share that with us. I run on an old core i7 8700 with 32G RAM and PI is really quite fast.

Some other thoughts: are you binning at all or are you at the full resolution? Are you using drizzle?

Andy, when I first started using PI it was with a lot of full-frame images and on a 16GB laptop. Integration took a seriously long time (maybe longer than 10 hours) as once it runs out of physical memory it spends most of its time swapping data to and from disk - in this case a hard-drive rather than nVME. Moving to a fast CPU with 64GB of RAM made an enormous difference!

John Noble:
Anyway bottom line I just can’t believe PI is 10 + times slower than DSS

It may be more accurate to say PI is doing 10x more than DSS. DSS has fairly simplistic algorithms and rejection techniques, and afaik no real normalization. If you want to do an apples to apples, disable everything in PI except calibration registration and stacking, and pick a simple rejection algorithm like linear fit and it will probably go as fast as DSS

John, it sounds like you need to buy a new computer. My five year old MacBook pro with 16G RAM takes around 30 minutes to do 100 APS-C frames from my 2600MC Pro on the highest setting in WBPP. …and I don't have to download any drivers… Sorry couldn't resist and old PC dis…

scott

Brian Valente:
John Noble:
Anyway bottom line I just can’t believe PI is 10 + times slower than DSS

It may be more accurate to say PI is doing 10x more than DSS. DSS has fairly simplistic algorithms and rejection techniques, and afaik no real normalization. If you want to do an apples to apples, disable everything in PI except calibration registration and stacking, and pick a simple rejection algorithm like linear fit and it will probably go as fast as DSS

Good to know but at least to my untrained eye I don’t see a huge difference in the final image. That said I only have a sample size of 1!

I will give all the suggestions a try because I’d rather work in PI as I said before I really like the workflow!

Thanks again

John

John Noble:
Good to know but at least to my untrained eye I don’t see a huge difference in the final image. That said I only have a sample size of 1!

You may not see a huge difference. It could be your data, your eye, etc. who knows

the point is, you don't have to turn on all that extra stuff if it doesn't make a difference to you. turning off = faster

I use an APS-C size sensor with 16 MP resolution and I use APP for calibration and stacking. In my opinion it gives at least as good results as WBPP. And it does it in a shorter time than WBPP. After that I bring the stacked linear image into PI and do the post processing there. In my opinion that is where the power and usefulness of PI is to be found, in particular due to all the third party scripts and processes that are available. My computer is an i5 processor laptop with 16 GB ram and SSD drive. It works fine for this workflow. And I do believe APP has local normalization that is similiar to PI. So I would recommend using APP instead of buying a very expensive computer, at least try it out on demo.

Also, in my opinion, it is really about time that the PI developers turn their attention to GPU based processing. This is beginning to be a standard technology in image processing today. It has the potential to really speed things up and it is really disappointing that PI does not already use this technology. There are other astro image processing tools, like Startools for example, that does this. It seems to me that the PI developers are just simply taking it for granted that their users shall buy extremely expensive computers to compensate what their software lacks in computational efficiency.

Frankly, if you want a fast calibration use Siril. It is way faster than PI and costs exactly nil/zero/nada/nothing. Unlike APP. And has all the calibration capabilities, in the main, that PI has.

John Noble:
I've been using DSS and Photoshop to process my images for the last few years and decided to give PI a try this week. I'm impressed with it and the wealth of resources out there to support the tool. The question I have is about the speed of the calibration steps.

I have a 16 GB RAM Surface which, using DSS, on a typical batch of 60 or so subframes from my ASI 6200 I'd expect to have them calibrated, cosmetically corrected, aligned and stacked within around 30 mins - all be it with quite a bit of manual intervention but still not an excessive amount of time. Testing PI yesterday I really liked the WBPP set up but when I loaded the files in and set it running even after 10 hours it was not done. I broke the package down into smaller batches say 12 files at a time and even then that was taking well over an hour to run.

I understand PI is more sophisticated and does bigger 'sums' etc but this duration is prohibitive - what I'm wondering is have I missed something or are you simply going to say 16 GB Ram how old school is that ;0).

I'd really like to shift to PI but the thought of dropping another $1500k minimum on a new laptop does not fill me with joy!

Many thanks

John

The largest batch I have run were 600+ subs of FF (QHY600) in PI. It took nearly 20hrs on a 64GB machine w/ 8 cores. When upgrading to a bigger camera one should always factor in at least the same amount for new computing hardware. To my experience PI produces better results than DSS at the cost of significantly higher computation time. But I think it is worth it.

When upgrading to a bigger camera one should always factor in at least the same amount for new computing hardware.

I don't think this should be necessary. We have a right to expect the producers of highly expensive software products like PI to continously improve their products, and this should include also processing speed and efficiency. When this does not happen we should complain to them instead of just accepting that we have buy more and more expensive hardware. As has been pointed out here there are other products on the market, some even free, that produces similar image quality at much higher processing speed, and this should also be pointed out to them. If we as their users just blindly accept that we have to compensate their poor development with our money, things will never change. If they ignore our legitimate requests for improvement we should move to other products on the market. If we don't behave as active customers PI will not learn what they should do.

Related discussions