A toast to PGGB, a heady brew of math and magic

[austinpop]

On 5/3/2021 at 11:28 PM, Zaphod Beeblebrox said:

I am open to implementing WavPack output format for higher rate files

 

Since the DAVE requires 16FS, which makes FLAC not an option, I decided to try WavPack as an experiment. I used dBpoweramp with the following settings:

I tried both Normal and Very High Quality. Both yielded a very nice compression of 65-70% over the original, while all tags were preserved.

 

Functional Issues

I used HQP in my normal "pass-through" mode of "filter=none, dither=none, DAC bits=32." Interestingly, I could drag and drop .wv (wavepack) files to the HQP window, and they played just fine. However, even after an HQP library scan, I could not "see" the .wv files, either via the Browse button on HQP, or via the HQPDControl app. To complete out the puzzling behavior, once I had dragged the .wv file to HQP, it was visible on the HQP playlist, AND HQPDControl could see it too. I need to report this to @Miska on the HQP thread. This was on the latest 4.10.3 Windows version.

 

Listening Impressions

I so very much wanted the wavpack files to sound the same as wav! But alas, I do hear a difference. Is it huge? No, and it still preserves most of the goodness PGGB brings. But if ultimate SQ is what you're going for, stick with WAV. If, on the other hand, storage is holding you back, then wavpack (and the already supported FLAC) could be a very good option.

 

Call to Action

Folks, please convert some wav files to wavpack (using tools like dBpoweramp), and test your music player for functional support, and gauge the sound quality. If enough players support it, it would incent @Zaphod Beeblebrox to add it as an option.

[austinpop]

55 minutes ago, mrkoven said:

if what @austinpop mentioned proves true (wavpack lowers SQ a bit) then thats unfortunate.

 

I really would like people to test this themselves! Your perception may be different than mine. Plus, some may be fine with a smidge reduction in SQ for 3x less storage. One size (pun intended!) does not fit all.

[austinpop]

45 minutes ago, Fourlegs said:

I will try today!

 

if my ears tell me right, I am in any case hearing better sound quality with my MPD playback instead of what you use so even if there is a slight trade off with wavpack it might be less noticeable. just kiddin’ 🤪

Yeah, every platform is different, with different trade offs and options. The K50 and Extreme are very different beasts. Curious to know what you hear.

[austinpop]

7 hours ago, ajm said:

Ending of Mahler 8, Solti CSO

 

An excellent album, and one of my all-time favorites! There is actually a 24/96 version that does improve on the Redbook. This is one of those rare albums where you want to use the "combine" feature of PGGB (https://www.remastero.com/guide.html#Combine). Since you have this album, you know that it has 16 tracks, but in terms of continuity, tracks 1-6 and 7-16 are contiguous. When I process this, I use the combine feature to combine these into 2 longer tracks. This does take more memory, but man, is it worth it! The resulting combined tracks 1-6 gets processed at 1029M taps, and tracks 7-16 (over an hour long) at 2478M taps.

 

But back to the demo. Scrounge around your computers... you sure you don't have one that has (or can be upgraded to) 16MB? If so, you can easily gargle-blast a short Redbook track like track 14 (Neige Neige) or track 15 (Blicket auf).

[austinpop]

 

3 hours ago, Fourlegs said:

ZB, using the Mscaler on pass through might of course be an interesting concept because it would enable feeding the hires PGGB files to the Dave via dual BNC which might have some advantages for the Dave instead of using USB. I guess that means it is time to experiment . . . . .

 

Nick, don't forget the HMS still has its own Amanero USB controller in the path, and if you're sending a 16FS stream to the HMS, then you're still incurring the same Amanero latency as the DAVE.

 

IMO, the simplest test, for people with HMS'es who want to trial PGGB, is to compare (let's assume a Rebook sample track (16/1FS)):

1. PGGB 32/16FS > USB > DAVE
2. Native 16/1FS > USB > HMS > DBNC > DAVE

This will tell you enough to decide if you want to explore PGGB further.

 

For those looking to dig deeper, I would submit that a battery of tests needs to be run to tease these concepts apart. Ideally, you would want to try all of these:

1. PGGB 32/16FS > USB > DAVE
2. Native 16/1FS > USB > HMS > DBNC > DAVE
3. PGGB 24/16FS > USB > DAVE (24-bit stream gets zero-padded to 32-bit, as Amanero requires 32-bit)
4. PGGB 24/16FS > USB > HMS > DBNC > DAVE (potential benefit of the HMS's galvanically isolated USB, but adds complexity, DBNC etc)
5. PGGB 32/16FS > USB > HMS > DBNC > DAVE (dual noise shaping, as HMS noise shapes PGGB 32-bit stream to 24-bit)
6. PGGB 32/16FS (no NS) > USB > HMS > DBNC > DAVE (some precision loss in PGGB, going from 64-bit float to 32-bit int with no noise shaping)

I did some of these comparisons early during the development of PGGB, when I was deciding whether to keep or sell the HMS. Don't get me wrong — I love the HMS, and ideally I would have kept it, but I was looking to mitigate the very large expense of buying my Extreme. So my results were:

• 1 vs. 2: clearly, I found 1 better, and it has continued to improve, which is why I sold my HMS.
• 1 vs. 3: it's been a while since I did this test, but for me, 1 sounded better. Since Amanero is forcing 32-bits at the USB boundary anyway, there seems to be a small SQ benefit to putting higher precision 32-bit data there, instead of zero-padded 24-bit data. 

I have never run 4,5, and 6, but look forward to results if others are motivated to give it a go.

[austinpop]

7 hours ago, chrille said:

Hello Rajiv, while I agree that the Solti Mahler is indeed a good vintage  DECCA analogue recording . But imho  modern hi res recordings like the ones from SSFO or LSO LIVE are better SQ wise than that one.

Regarding modern digital HI RES  like for example the BIS Sibelius symphonies you mentioned in one of your reviews or the imho very good Indian classical Vijay Gupta album you also used, could you indicate on a percentage scale the differences improvements as you hear those two examples PGGB´d compared to mscaled?

In other words if you have dumpted your Mscaler PGGB is better in your opionion right?

 

But how much better? 5 -10% or more?

 

Hi Chrille, 

 

I find PGGB to improve everything — from the crappiest recording in my collection to the best. In general — and I'll defer to @Zaphod Beeblebrox to weigh in if this is predicted by the math — the improvement is greatest with Redbook albums, and smallest with DXD. Stated another way, the greater the upsampling multiplier, the greater the apparent improvement. Here are a few examples:

• Redbook: Haydn Creation (Arkiv, mid-80s) (16x multipler)
  - I love this performance, but never really thought of this as a great recording. PGGB has just transformed this album.
• 24/96: the aforementioned Sibelius BIS recordings by Osmo Vänskä and the Minnesota Orchestra (8x multiplier)
  - These are already-great modern recording, but improved even more with PGGB. Are they "transformed?" Perhaps not, but I would never go back, having heard the PGGB version.
• 24/352.8: The 2L Arnesen Magnificat album (2x multiplier)
  - This DXD version is considered reference quality already. Yet, PGGB improved it in important ways. 

All that said, a PGGBed lower-res mastering does not leapfrog a PGGBed hi-res mastering, although the delta between the 2 PGGBed versions will certainly be smaller. Anyone who wants to try this for themselves should just go to the 2L testbench and try the experment with some of the samples there.

 

 

7 hours ago, chrille said:

I have so far only heard a native  16/44.1 PGGB ´d to 32/705   186M and would like to hear what PGGB has to deliver with hi res before investing in a better laptop.

But I can play the testrack both in Audirvana and Pure Music on my old MBP.

 

Check your PM.

 

As regards PGGB vs. MScaler, this is really something you should evaluate for yourself. I have always asked that readers of my reviews and postings not use my findings as a basis for making purchasing decisions. At best, my findings are a guide, a suggestion to explore for yourself to try, and then let your ears be the ultimate judge. What is "transformational" to my ears on my system may be "hm, nice, but not amazing" to your ears on your system.

 

I actually wish I had kept my MScaler for use with streamed Qobuz music. Since you don't stream music, your decision may be simpler. If you find PGGB to your liking and sufficiently superior to using MScaler, then the sale of the latter could easily fund the purchase of a PGGB license, a kickass system for PGGB processing, extra storage for your music server, and still have a nice chunk of change left over.

 

You just have to hear it for yourself and decide for yourself. 

[austinpop]

That’s great to hear.

 

15 minutes ago, 3ggerhappy said:

to dxd 32bit 81M(should be 24bit)

 

Yes, you should noise shaper to 24-bit for dCS.

 

15 minutes ago, 3ggerhappy said:

and I am already liking what im hearing with the dCS Rossini(too bad I cannot bypass the filters. Still...).
 

 

I was under the impression that the Rossini would bypass the internal filters when presented 24/8FS. Isn’t this effectively what happens if preceded by the dCS Upsampler?

[austinpop]

@ambre

 

Please take a look at ZB's PGGB Guide here: https://www.remastero.com/guide.html. The section "A note about software volume control:" covers the pitfalls you could encounter!

[austinpop]

7 minutes ago, muski said:

Hi —

 

Really enjoying this, but lots of questions!

 

First, I'm getting static in the right channel when I play back 768kHz content from Roon via HQP. Doesn't happen with 705.6kHz content. Any ideas?  (HQP settings screenshot attached)_

 

Second, is the only way to launch it on the Mac is from the Terminal? Like this:

/Applications/remastero/PGGB/application/run_PGGB.sh /Applications/MATLAB/MATLAB_Runtime/v99/

 

 

 

I suspect the culprit is "auto rate family." I don't use the Embedded version, but the equivalent on the Desktop version is "adaptive output rate."

 

Doesn't it need to be checked? @Zaphod Beeblebrox 

[austinpop]

A few PGGB findings on DSD albums and provenance in general

 

In my experience, and as many of you have discovered, PGGB really makes a fantastic effort (it can only do so much) recovering the transient accuracy of DSD albums in your library. This has proved invaluable for recordings for which the only available native formats were DSD or inferior-sounding Redbook masterings. In my collection, a prime example of this is Kaplan's Mahler 2nd.

 

However, for others, the choice is not so clear. Many SACD/DSD releases are:

1. Recorded in DSD or PCM
2. Converted to PCM (to rates up to 352.8kHz)
3. Mixed and Edited in PCM
4. Mastered to various DSD and PCM release formats.

Step no. 3 is critical. Once you've brought the data into the PCM domain, and given PGGB is designed to output PCM data, it's just logical to expect a PCM release to sound better than a DSD release, which has to undergo another conversion:

• DXD to DSDn, where n = (64, 128, 256, 512).

And indeed, I am usually finding — I say usually, as there may always be the odd exception to your ears – that it's preferable to buy (and then PGGB) the best possible PCM mastering. Here's an example:

 

When I bought this at nativedsd.com, I got both the DSD128 and DXD versions. While PGGBing the DSD128 results in an impressive no. of taps, it is the PGGBed DXD version that sounds the best (accurate, transparent) to my ears.

 

Pay attention to provenance, and for DSD albums, be on the lookout for those casual references to "edited and mixed in DXD." That's a hugely important factor!!

 

There are indeed some albums that were recorded, mixed, edited, and mastered all in the DSD domain. But they are few, and none of these have found their way into my "albums I love and must own" list.

 

What I'm doing

 

While the above may be good to know for future purchases, what to do with your existing collection? Like many, I have hundreds of ripped SACDs. For some that are my most treasured, I've tried to dig into provenance, and buy PCM masterings instead. For example, Michael Tilson Thomas' Mahler cycle with the San Francisco Symphony released on SFSMedia. I can't find a lot of provenance info on these recordings. However, they are now available at Qobuz in Hi-Res PCM formats ranging from 24/44.1 to 24/96. I ended up buying all of these in hi-res PCM, as they are notably better PGGBed than the ripped DSD64 versions from SACD.

 

How far you take this is up to you, but I'm looking at my entire library with a clearer eye, and making strategic decisions as I go along, especially for recordings that I particularly treasure.

[austinpop]

2 hours ago, CNoblet said:

Using a 2015 MacBook Pro (16 GB 1600 MHz DDR3 RAM; 2.5 GHz Quad Core Intel Core i7, Big Sur 11.2.3) I encounter no problems upsampling DSD64 to the PGGB default configuration of 705.6/768 kHz sampling rate, Auto 512 m taps, 32 bit output depth and auto adaptive noise shaping.  The only issue is with track lengths over 20 minutes, for which I change the output sample rate to 352.8/384.

 

The average DSD album takes around 10 to 12 hours to convert, so I run one DSD album overnight and multiple pcm albums during the day, using the trial version of PGGB. 

 

Playback with a bog standard headless 2011 mac mini running Roon or Audirvana 3.5 into a Chord Mojo is mind-boggling -- absolutely smashing.  My ears have arrived at their happy place.  I will purchase PGGB as soon as I can do so comfortably.  The best news (for me) is that I find it a complete cure for DAC upgradeitis.  Goodness gracious what a wonderful toy this is.  Thank you to Zaphod Beeblebrox and everyone else involved!

 

That's great to hear!

 

BTW - just as a teaser to you and others here who are debating what machine to use for PGGB. See below for what a fast machine with 128GB buys you. This is Mahler 5th (Fischer/Budapest) in DSD64. Notice the track lengths were long enough to exceed 3 billion taps, yet the entire album processed in 2.5 hours.

 

 

So yes, it is possible to make do with 16/32/64GB or RAM, and this makes perfect sense when you are evaluating PGGB. But once you've decided that PGGB is for you, and if your tastes run to large symphonies, with movements exceeding 30 mins in length, and these are on native formats like DXD or DSD64/128/256, do yourself a favor and get a robust enough machine with 128GB (or even more) RAM. You'll thank me later.

[austinpop]

3 hours ago, davide256 said:

Done the range of 176-353-705 rates and 24/32  bit depth in wav with my test track. Puzzles me that I can hear more with Stylus downsampling but details are

better enunciated with 705>176 than they are with native 176. I cache all music to a FEMTO clocked NVME drive + the IRIS uses a FEMTO clock. DAC tested

with was NOS Metrum Octave.... the Mojo internally up-samples PCM to 705/768 which is its own can of worms.

 

Dave, 

 

It seems like you're juggling with competing forces here.

• The Mojo is a Chord DAC, and to get the benefit of PGGB, you need to bypass the WTA1 stage - just like we are doing on our DAVEs, TT2s, etc. This requires 16FS PGGB files. You can't really use half measures here. Upsampling to 4FS or 8FS does not get the WTA1 stage out of the way, so the benefit of PGGB, while still possible, is going to be unknown.
• Yet I sympathize that the Mojo uses suboptimal connectors like micro-USB or 3.5mm coax as the only inputs capable of accepting 16FS. This does impose degradation due to poor cable choice and crappy connectors.
• And like many chains, a DDC has advantages to allow reclocking, regeneration, so I can see why the Iris was attractive - but only for native rates. For 16FS, you may be better off with a USB regenerator like the tX-USBultra or ISO-Regen. But this does add cost - I get that.

If this were my system, my inclination would be to first go the best route for signal integrity, which is to preserve 16FS end to end:

• NUC/Euphony/Stylus > PGGB 32/16FS file > USB A to micro-B > Mojo
• The manual for the Mojo claims to allow support for 16FS on the 3.5mm coaxial input. The excerpt from the manual is:
  • "1 x 3.5mm COAX SPDIF capable of playing 44.1KHz to 384Khz PCM (768KHz special operation) and DSD64, DSD128 in DoP format"
  • EDIT: the manual appears to be incorrect, as I found a post by Rob that confirms dual-SPDIF mode is not supported on the Mojo.
• If the above is truly an option, you could try the Audiowise SRC-DX adapter, and do this:
  • NUC/Euphony/Stylus > PGGB 24/16FS file > USB > SRC-DX > DBNC to 3.5mm cable > Mojo.
• The SRC-DX route, which admittedly also adds cost, would also potentially uplift SQ by avoiding the Amanero USB controller.

It just seems like the Iris, with its 4FS limit on the input, is not a great fit with the Mojo/PGGB path.

 

All that said, these are competing tradeoffs, and the path you're on may well get you to the best compromise possible, given the limitations. At this point, your own experiments are going to be the best way forward, as none of us are using (or would use) PGGB in this way.

 

I hope you find a good answer.

[austinpop]

30 minutes ago, davide256 said:

I hear you... am I correct that a Denafrips Pontus would not support 16FS?

 

Pontus architecture description

"True balanced 24BIT  R2R +  6BIT DSD (32 steps FIR Filters)"

 

 

15 minutes ago, Zaphod Beeblebrox said:

According to the specs on their site The Pontus can do up to 32FS via USB

 

 

But note the following on the Iris specs:

 

 

So you could send 32FS to the Pontus II via a direct USB connection — assuming your player supported 32FS. But if you put an Iris in the path, then you're limited to 16FS.

 

Also, if you follow the discussions on 32FS DACs in the HQP and Holo May DAC threads, getting 32FS to work over USB can be fussy, and requires specific drivers/firmware. And — because the Pontus II is an R2R DAC, your best results may come when you noise shape the sample depth to somewhere in the 18-24 bit range. Let's assume 20-bit is the sweet spot.

 

When the time comes, and you have the Pontus II in your hot little hands , you would want to try these paths and see what sounds better:

• NUC/Euphony/Stylus > PGGB 20/32FS file > USB > Pontus II
• NUC/Euphony/Stylus > PGGB 20/16FS file > USB > Iris > I2S > Pontus II.

[austinpop]

As ZB has pointed out, the Max Taps setting in the GUI is only an upper bound, and only kicks in when your track length reaches the relevant threshold. Refer to the table in the website:

 

 

 

If the GUI is suggesting a max setting of 256M due to your RAM size, then note that this has no impact if your track lengths are under 6:03 (at 705.6kHz) and 5:33 (at 768kHz). For tracks under these lengths, the filter length will be some value less than 256M that corresponds to the track length. Here's another table to illustrate. So if your track is only 5 mins long, it'll only use a 212M tap filter (at 705.6kHz).

 

Track Length (mins)	Sample rate (in kHz)	Taps to cover track (million)
1	705.6	42
2	705.6	85
5	705.6	212
10	705.6	423

 

On the other hand, for a 10 min long track, PGGB will limit the filter length to 256M due to the limited RAM available. If you had more RAM, it would have constructed a filter of 423M length.

 

I hope this clarifies things?

[austinpop]

3 hours ago, happybob said:

I'm wondering how much difference in sound quality (subjective I know) to 16FS with a Dave DAC is there between 256M taps and maximum number of taps (if I had a machine with 128GB RAM) for 44K/16 originals? 

 

There is certainly a difference. As I pointed out in my last post, the magnitude of difference will be greater if the track length is greater. Take the example of a 10 min track. It would have processed at 423M taps if you had had more RAM, but instead processed at 256M due to your limited RAM. The difference between a 423M and 256M taps filter would be quite minor. But if you had a 48 min track, which would have processed at 2048M taps with adequate RAM, whereas it processed at 256M taps on your system, the sonic delta between the 2048M tap-processed file and the 256M tap-processed file would be noticeable.

 

So keep that in mind - a lot depends on track length. 

 

If your favored genres are manifested as primarily 16/44.1 with average track lengths of 5 mins, you may not ever need more than 16GB to be perfectly content with PGGB. But if you listen to classical or other genres where long track lengths are common, investing in more RAM is very worthwhile.

[austinpop]

1 hour ago, happybob said:

I'd like to avoid doing multiple rounds of upsampling my library - so what I might do is to only process the albums with shorter tracks and then wait for those longer ones till I get my new system

 

As someone who just redid almost 300 albums because I switched from 32-bit to 24-bit files (for using the SRC-DX bridge with my DAVE), redoing the upsampling is not that big a deal. If you wait for a *rumored* machine, you're just depriving yourself. Even at a cap of 256M taps, PGGB makes a massive improvement over the native file!

 

BTW - during the initial development, I must have redone my albums 4-5 times as ZB made incremental improvements. I did not regret it one iota.

[austinpop]

1 hour ago, jeti said:

Could you comment on SRC-DX vs  32-bit USB on Dave? I guess that many people will be interested in this. 

 

1 minute ago, Progisus said:

Thanks. On a TT2 what is the advantage going dual bnc over usb? Assuming one has the src-dx already. My usb is from an EX.

Please see the section on USB improvements part 2:

 

 https://audiophilestyle.com/forums/topic/30376-a-novel-way-to-massively-improve-the-sq-of-computer-audio-streaming/?do=findComment&comment=1130590

 

[austinpop]

ZB likes his Gargle Blasters with extra Fallian marsh gas, and a twist of Arthur's towel. 

[austinpop]

9 minutes ago, sig8 said:

Anyone experiencing 32 fs PGGBs file buffering? My i9-10900, 32 GB PC is directly connected to Terminator+ DAC, and it is buffering like crazy. I am using Audiolinix and HQPe.

 

Files are stored on internal NVMe.

 

 

What are your DAC Bits and Buffer Time settings in HQPe?

 

Edit: easiest if you paste a screenshot of your HQPe settings.

[austinpop]

3 hours ago, sig8 said:

DAC bits 24, buffer; I tried 250 and 0, both cause buffering.

 

I will defer to the AL experts - I haven't used AL in over 2 years. I do seem to remember Piero enabling some systemwide ALSA config options - not sure if that is even relevant now.

 

Anyway, your best bet would be to seek advice on the AL thread, mentioning you're trying samples rates of 1411.2/1536 kHz.

[austinpop]

32 minutes ago, R1200CL said:

It’s my understanding after just after reading a few posts, that a powerful computer is needed. Now, assuming this is more or less a one time operation, I was wondering if anyone had looked into the option of renting a server in the cloud ? Is this a way to go ? And if not, why ?

 

Standing up a VM in the cloud with the necessary resources is pretty straightforward. The big issue is the cost (in time and $$) to move native files to the VM (ingress), and huge upsampled files from the VM back to you (egress).

 

That said, never say never, this is being looked at!

[austinpop]

There seems to be some ongoing confusion about what bit depth to process PGGB files for the DAVE/TT2/Qutest when going through the SRC-DX USB-to-DBNC bridge.

 

The short answer is 24. (Not 42. That's the Ultimate Answer. We aren't there yet. The Galactic Government gave us a waiver. )

 

Why? TL;DR below.

 

When processing PGGB files, the prime directive is to do so in a way that the PGGB files are not molested on the way to the DAC. So you want to look at the bit depth allowed by the eventual DAC interface you're driving, not the SRC-DX USB interface you're initially driving. WIth the SRC-DX, you're eventually driving the dual BNC inputs on the DAC, and these only support 24-bits.

 

You might think: but the USB input on the SRC-DX supports 32-bit, so why not use 32-bit? The reason is that then you're asking the controller chipset in the SRC-DX to do the 32-to24-bit conversion, and we don't know what it does. Does it truncate? Round? Dither? None of these are good options because they all negate the noise shaping that PGGB did for you so carefully. The noise shaper in PGGB is a critical contributor to the superior SQ, and molesting the signal this way will affect SQ.

 

There are many other considerations when picking bit depth, and they vary by DAC and connection path. The above was specifically addressed to Chord DACs being driven via an SRC-DX bridge.

[austinpop]

3 minutes ago, ajm said:

In an earlier post, Rajiv @austinpop, you mentioned that 256GB RAM might be beneficial for PGGB. As most PC's are not upgradable beyond 128GB, is it possible to hint what 256GB might be needed for (is it just very long tracks?) Could @Zaphod Beeblebrox perhaps also comment on whether any future developments of PGGB are likely to further increase the RAM requirements above 128GB.

If one is purchasing a new PC largely for the purpose of PGGB activity, it would be particularly frustrating to find in due course that the chosen machine was actually inadequate because the RAM could not be expanded sufficiently. In addition, machines suitable for more than 128GB RAM are generally in a significantly higher price bracket.

 

I can't recall what I said, but my intent was to say that if you have 256GB, it can speed up some very challenging remasters, like a DSD256 track that's 45 minutes long, with Max taps set to 8192!

 

But you don't require 256GB, or even 128GB for that matter. I have a 128GB machine, and I have yet to run into an album that won't process. Some do take a very long time, but again, this in only for those pathological DXD or DSD albums with long track lengths. They do get done, eventually.

 

The key thing to keep in mind is that PGGB needs a certain amount of virtual memory to run, and you must accommodate this virtual memory by configuring your system appropriately. To explain, here is what Task Manager shows on my system. The amount of available virtual memory for the entire system is shown in the Red oval.

 

 

This number is actually the sum of the physical RAM (in my case 128GB) and the allocated paging files, shown below:

 

 

By human arithmetic, that total 128+350=478GB, but math was never Windows' strong suit  so they show it at 470GB. Good enough for government work, I guess.

 

So what this is saying is that as long as all processes in your system are allocating less than 470GB of virtual memory, things will fit. When running PGGB, you really want to only be doing that, so this is effectively the virtual memory that PGGB can use. If you get an OOM (out of memory) error when running PGGB, it is telling you that it has exhausted virtual memory. You can remedy this by increasing the paging file size, as long as you have free space on the disk on which the paging file lives.

 

But keep in mind that you still only have 128GB of RAM, and this is where paging enters the picture. The amount of virtual memory that is allocated (not necessarily used) by all processes in the system is known as the committed memory, or the commit charge, in Windows-speak. This is the amount in the green oval. If this commit charge grows to exceed the available virtual memory, you get OOMs.

 

Task Manager also shows you the amount of physical RAM that is in use (the purple oval). What you will see if you observe Task Manager during a heavy PGGB run is something like this:

• Commit charge will start growing
• Memory in use will also start growing, but won't be as high as the commit charge. This is because an application may commit (allocate) virtual memory, but not necessarily use it. The amount of virtual memory a process is actually using at any time is called its working set, and you can observe this with other tools like Process Explorer. But let's go back to the application's progress
• If at some point, the memory in use grows to fill the available RAM, the OS will start paging. This is a mechanism where because the demand for physical memory exceeds what's available, the OS will temporarily expel pages not in use to accommodate pages in use. These pages are written out to the paging file on disk.
• When this happens, it's called paging, and it can be observed in multiple ways. In the Task Manager view, you will see a burst of I/O to the disk, so the disk utilization (in the blue box) will go up. 

This is why having the paging file on a really fast SSD is such a good idea. In days of yore, for those old enough to remember the days of HDDs, a system that went into heavy paging essentially became unusable. With modern SSDs, and with a batch-oriented application, you can actually tolerate a moderate amount of paging, but the inevitable slowdown in execution time can be mitigated by ensuring the page file is on a fast SSD.

 

This is why it is strongly recommended that you run PGGB on a machine with a fast SSD hosting the OS and the paging file.

 

Summary

 

While PGGB will run much faster when you provision your machine with more RAM, you can still run even the most challenging workloads, as long as you supply adequate page file space. Make sure to put that page file on as fast an SSD as you can manage, to improve run time.

[austinpop]

Before anyone asks — the rationale is the same on MacOS, but I'm not intimately familiar with how MacOS manages virtual memory. 

 

I do know that MacOS doesn't rely on preallocated paging files, but dynamically generates one (or more) swapfile's in dynamic response to virtual memory demand. How well this works is to be determined, although I have heard some reports from PGGB beta users that suggested that this dynamic allocation strategy, while more robust and elegant, does seem to be slower. 

 

This is perhaps not surprising, as the advantage of the preallocated pagefile strategy on Windows, while inelegant, does greedily allocate paging space ahead of time, which may be a run time advantage.

 

In any case, minor differences in run time are irrelevant if you're using PGGB the way you should — as a batch processor, that you set up and let run to completion in the background.

[austinpop]

9 minutes ago, muski said:

I have a 2020 iMac with an 8-core i7 with 64GB RAM and a very fast SSD (so swapfiles should work really well).

 

How much free space is on the SSD? For swapfiles to be created, there must be adequate free space on the SSD. If there's no free space left, then the OS can't grow the swap space, and hence will throw an Out of Memory error.