WBPP problem

Maybe better to post this in the pixinsight forum directly? 

(Wow, 96gigz of ram 🤯)

Big data is big data.

You have a large dataset and you are using LN and probably the ESD rejection algorithm. You can do a back-of-the-envelope calculation if 1547 9576 x 6388 XISF files can fit to RAM (even if you think 96 GB should suffice). And on top of that there is the memory footprint of the rejection algorithm used… plus the memory footprint of Local Normalization. You can switch off LN and choose a simpler rejection algorithm (Winsorized Sigma Clipping for example), or just use FastIntegration 

I experienced a similar problem when stacked twice 1100 subs. I can advise you to restart the PC just before running the PI and WBPP and to avoid running anything else. Usually, I hibernate my system, but I found it ends with a much smaller amount of available RAM than I expected. I have 64GB RAM. Look at your screenshot, there is only 25(ish) GB RAM available!

Sergey Volkov:
You can't eat an elephant whole, but you can cut it into pieces. Break the problem into two, three parts, then put the image together from the stacks

This.
I used to have similar issues with 600-700 frames already but more than 3000? I got a bit further by messing with the windows swap file settings but you never know if it will crash or not. I believe it is much easier to follow Sergeys advice. Nowadays I simply stack each night, then combine with star alignment and image integration. I think the minimum you need is 3 sets for image integration.

Vitali:

Dan Brown:
Ryzen 9950, 96 gig ram

Try to reduce the number of threads that PixInsight is allowed to use: EDIT -> Global Preferences -> Parallel Processing and Threads -> Maximum Number of Processors Used. Also uncheck 'Enable thread CPU affinity'.

By default PI will use 32 threads on your CPU. You should set the option to 16, or even lower to 8. I have a Ryzen 5950X with 64Gb RAM and setting the value to 16 helped in my case when integrating 1000+ frames.

It would be interesting to know if this will also help with your data.

Vitali

Is it going faster this way or just doesn't stuck?

I had a similar issue (with similar hardware) with my andromeda project which sits at 1500 frames over 9 nights.  I had *enabled* CPU affinity, which did cause PixInsight to lockup. Going back to default settings saved the day.

As others have said, you may be pegging your core temp too high.  I have a similar system, although only a 7950X (still 16c/32t).  I "only" do OSC data and "only" APS-C sized, but did a WBPP integration of somewhere near 1100 subs.  It took a while, but got through it fine.

You don't mention anything about your cooling, whether it's air or liquid-based.  I do have an AIO cooler on mine and while it runs pretty darn warm while integrating and doing LN, it's never had an issue.

Somebody asked if the whole data set can fit in memory for LN and rejection.  To my knowledge, the whole data set doesn't need to fit into memory.  The data gets processed and compared in chunks.  Others have mentioned cutting back on the threads - FWIW, I don't cut back at all.  WBPP manages the threads and RAM well.  They've also mentioned the swap files.  While a good idea to make sure they're set up and tweaked right, to my knowledge, they're for while you're doing post-processing and don't come into play during WBPP.  I do recommend the suggestion of rebooting the PC and running nothing else before stacking a large data set.  I've found that Windows can grab memory and not let it go before.  I believe disk access can even be affected by Windows.  A clean start definitely can't hurt…

Of course, I could be wrong on any of this except for my own experience…

Hi, have you checked the log that WBPP creates as it runs through its steps? It's in a sub folder of the WBPP folder where the images are being saved. There might be expanded data there on what exactly it is hanging up on. IMO if the CPU is just sitting at 7%, it is not thermal throttling, but WBPP just gave up/crashed at some point.

Hello, I am no sure, if I am right: The CPU has 32 threads, this means it would be possible to address 128GB of RAM. If the RAM is not enough, windows uses the so-called pagefile.sys to swap data, which are not used at this moment. If you will stack so many pictures, I think the available RAM if not enough for this and harddisk space is used. I am using APP to stack pictures and already stacking of about 60 pictures needs 25 GB of harddisk space. I would try to increase the allowed size of the pagefile.sys.
I am using an Intel7 (8Threads) with 32 GB of RAM and I have already stacked about 200 Pictures of a Full Frame Camera (Nikon Z6). My pagefile.Sys is manged by Windows and I have set it on a SSD. However it takes a lot of time.

Thank you all for the input. I will keep you posted on my progress.
I'm using a liquid cooler, Corsair iCUE H150i ELITE CAPELLIX 75 CFM Liquid CPU CoolerI just downloaded software to monitor the cpu temp. 
Dan

Good possibility. I believe that 95 deg. C is the max temperature for your cpu.