WBPP stacking duration

Wow…you have to work pretty hard to achieve a nearly 5 hour stacking job-even with a 2012 Mac!  

I can’t talk to you about the algorithms but I can tell you one big thing that’s causing it to take so long.  Using 60 flats, 60 dark flats, and 30-40 dark files is a complete waste of time.  I don’t have it here on my iPad, but in other threads, I’ve posted a chart showing the impact on noise as a function of the number of calibration files derived from the statistics of a stacking calibrated files.  For 48 lights, using calibration masters containing much more than around 16 subs doesn’t provide any significant improvement in SNR.  You didn’t say how many different exposures you are using or the exposure lengths for your flats so it’s hard to parse out how you could further consolidate the process.   Flat darks aren’t always needed—but that depends on how you are taking the data.  It depends a little on how you operate, but in most cases, it is MUCH faster to create reusable flat, dark, and bias masters prior to stacking your lights.  It those cases, you save time by stacking the calibration files just once; rather than stacking them with each new image.  If you work a little smarter, I suspect that your old Mac will have no trouble stacking a set of 48 lights in less than 10-20 minutes—even with the ASI2600 using 1x1 binning.

John

As above:  did you use Local Normalisation?  On my fairly old workstation it tends to double my overall integration time and quite often produces worse results than if I didn't use it.  I also stick with master darks and master dark flats for several months and as such that part of the process is zero seconds.

FYI:  I do use WBPP most of the time, but tend do it in two halves (calibrate the images -> SubFrameSelector to choose best images -> Back into WBPP to register and integrate)

Personally, I would not recommend an Apple laptop for PixInsight.  You'd be better off with an Nvidia-based gaming PC with multiple NVMe drives (based on their plan to start taking advantage of CUDA acceleration).  Even things like Starnet++ run so much faster using CUDA.

by default WBPP tends to automatically select ESD clipping for pixel rejection. this algorithm is extremely memory and cpu intensive. this could lead to paging (and extremely poor performance) as Roger explains above. if you manually select Windsorised Sigma Clipping that part should be a little more quick.

and as others have explained WBPP is also now using Local Normalization by default, which isn't super processor intensive but it is another step that has to be run against each subframe, which programs like DSS, etc. don't do. depending on the number of subframes this can also increase runtimes a lot.

rob

Almost 2 hours of that was due to local normalisation … I'd suggest you really need to check if you need to do that based on your subs.  I know the WBPP script added it as a default recently, but that doesn't mean you have to use it.

I also think that doubling your RAM would help a lot.

Hi,

you may run the PixInsight Benchmark to check how your machine is performing.



PI is fairly IO-heavy, so the speed of the drive where you keep your files is important for overall performance. My very first image integration with PI took 8+ hours because I was stupid enough to run it from a  slow NAS. In addition you should create swap files - 1 for each core of your cpu under Preferences





Using different drives for multiple swap files is better than what I'm doing ...
Your files and the swap files should be located on SSDs preferably nvme ssds. With a lot of RAM a RAM-disk is another good solution.

PixInsight — PixInsight Benchmark provides you with typical results for various setups.

Clear skies
Wolfgang

the thing about swap files though is that i don’t think they are used at all by any process in the WBPP pipeline. they are the backing store for the view undo history. 

wbpp speed is more about putting your subs (and the working diectory) on fast storage since just about every task in the pipeline is reading images from disk and writing them back out. 

i couldn’t detect a meaningful difference between running wbpp on subs stored on an nvme ssd vs. a ram disk on my M1 ultra. but this is probably because that machine has a ridiculous amount of bandwidth available in the storage subsystem.

Daniel Arenas:
@John Hayes I take flats 5 seconds time exposure. That’s the reason for taking darkflats. That and because in other posts most of you told me that bias are not needed in dedicated càmeras and darks also includes the electronic noise that are recorded in bias.

Hi Daniel,
At 0C, the IX455 only produces 0.014 e-/pixel/second of dark current so at 5 seconds you don't need dark flats.  Dark flats are just increasing your computing time with no benefit.

You are correct that dark data also includes the bias offset; however, the problem is that in order to properly subtract the pure dark signal, you have to be VERY careful about even small offsets.  It is so easy to subtract bias that I still recommend doing so--just to avoid the possibility of a calibration error.  If you are absolutely certain that the bias signal is identically zero across the chip, then it is safe to ignore bias but that's almost never the case.

Here is the chart that I was referring to.  With 50 subs, 64 subs in the calibration master only provides an additional 9%-10% reduction in noise compared to using only 16 subs in the master.  That small gain incurs a large penalty in computing time.   I suggest trying a calibration with 16 subs and I'll bet you that you will have a very, very hard time seeing that additional 10% noise reduction.


John

Andy Wray:
@John Hayes You are obviously right about the dark/flat calibration stuff.  Daniel's real problem is, however, with the time it's taking him in registration, local normalisation and integration.  Not accepting the WBPP defaults in these areas and maybe leaving local normalisation out of that process (given his limited PC capabilities) would actually cut hours from the process.

I have not used Local Normalization in a long time, as it can cause more problems than it solves. Do folks really see that much benefit from it? If it is adding hours to the process of stacking images -- I would hope there is some tangible benefit of doing so.

Hello everyone!

After having spent some days disconnecting from the routine I'm here again. I want to let you know that I'm very gratefull about your help and comments and I very appreciate them.

one thing i do see is that the LN reference seems to be composed of all 58 of your frames. i don't know if that's a display bug or if you are running an older version of WBPP, but the tooltip in mine (1.8.9-1 with WBPP 2.4.5) says that it will integrate the top 3-20 frames for an LN reference. since the LN reference generation times and master light integrations seem to be taking the same amount of time, it really does seem to me that WBPP is integrating all 58 images as the LN reference.

Hi @pfile,

That's a parameter that I just click the checkbox. I do not force to use all the pictures in the configuration. It's something that WBPP just do it. I show you a screen capture with the parameters' configuration. I'm using the latest version of WBPP 2.4.5



In presets I choose the best quality. If the price is only the speed of stacking I'd rather to pay it and have a master-light with the best quality.

Hi,

you may run the PixInsight Benchmark to check how your machine is performing.

Hi @kuechlew ,

I have done it but I don't Know how to interpret the report given.

Here it is, from my Mac.

Andy Wray:
just thought I'd give you something to compare against.  This is using my 4 year-old Windows workstation with the following specs:

Core i7 8700 3.2GHz 6 core
32G RAM
Two NVMe drives: one for my boot and another for my data.  I've created separate swap files on the two drives.

The below is a slightly larger number of subs than yours being measured, registered and integrated.  That said, my camera has a smaller number of pixels.

Hi @Andy Wray,

So, you have 2 more cores and with more GHZ (3.2 in front 2.3). The processor is an i7 but I supose that this model number 8700 indicates that have much more features (whatever that means) than mine wich is 3615QM.
And also you have 16 Gb of RAM more than me and I supose that they are DDR4 with good speed.

I think It's clear that all of this bunch of features move better the data. Ah! My SSD it's not an NVMe, it's one 10 year-old. Just 18 minutes is impressive!

John Hayes:
Hi Daniel,
At 0C, the IX455 only produces 0.014 e-/pixel/second of dark current so at 5 seconds you don't need dark flats.  Dark flats are just increasing your computing time with no benefit.

You are correct that dark data also includes the bias offset; however, the problem is that in order to properly subtract the pure dark signal, you have to be VERY careful about even small offsets.  It is so easy to subtract bias that I still recommend doing so--just to avoid the possibility of a calibration error.  If you are absolutely certain that the bias signal is identically zero across the chip, then it is safe to ignore bias but that's almost never the case.

Here is the chart that I was referring to.  With 50 subs, 64 subs in the calibration master only provides an additional 9%-10% reduction in noise compared to using only 16 subs in the master.  That small gain incurs a large penalty in computing time.   I suggest trying a calibration with 16 subs and I'll bet you that you will have a very, very hard time seeing that additional 10% noise reduction.


John

Hi @John Hayes ,
That's a great discussion. With my non modified DSLR I used to take bias and flats with 1 o 2 seconds of exposure.
When I get my ASI 2600 MC Pro 16 bits dedicated camera I asked what was the best way to proceed and someone, which opinion I appreciate as evereyone's, told me that dedicated cameras don't need to take bias and that is better to take dark-flats If I took flats from more than 1 or 2 seconds exposures. I use a flat pannel and with the NINA's flat assistant the best value is 5 second exposure, so I did dark flats.

What I don't think to have known much about is what you say from the calibration master. I mean, I have undestood about the graphic explanation but this 6 subs mean darks? flats? bias? dark-flats? or 16 of each?

To take bias in NINA with a dedicated camera you set the time in 0 ?

I have not used Local Normalization in a long time, as it can cause more problems than it solves. Do folks really see that much benefit from it? If it is adding hours to the process of stacking images -- I would hope there is some tangible benefit of doing so.

Hi @Bill Long 

I'm a Newbie on this with just a year of experience, maybe it's my 5th stacking with WBPP, and that's the first with fits taken with a cooled dedicated camera. And moreover with data of two night sessions. I check in the parameters what it's supposed to be checked but I'm not in the point to know how to interpret all processes of the stacking in a technical way just to know what are the pros an cons of each step or if it's better to avoid one or another in that or other cases. I assume that I'm still learning and I have a long path in front of me.

LN was kind of a mess for a long time and usually created more problems than it solved. but then normalize scale gradient came along and lit a fire under juan… and he rewrote LN. i think it uses photometry now but not 100% sure of that. anyway it is a lot better behaved now. 

for whatever reason it is now on by default in WBPP and new users probably are not aware of this.

Maybe that means that admins believe that the update of the internal process / algorithm (I don't know what's the correct word) of LN has solved all the problems and It will give more pro than cons.... I'm just guessing.  I didn't know that LN gave so many cons before.

Well I have done a test and I copied all the data from my external thunderbolt HDD to the internal 10 year-old SSD. I have better stacking times. In front of the 4 hours 49 minutes, now I had 3 hours and 29 minutes

That are the 2 screnn capture. First the new one, and then the old.





I'm going to try to do that again with my daughter's windows laptop that it's not a powerful one butmuch more modern than my Mac.

What do you think to use a Gaming PC (not laptop, a desk one)  for using PixInsight? My teenage daughter ask me to buy one to play computer games and maybe it could be a good thing for both.

Well, in order not to mix things, may be it will be better to create another thread to talk about that.

Thanks everyone and don't hesitate to let me know you opinion / experience or any kind of contribution to me and to all these great astrobin's community.

Kind regards and clear skies!