A toast to PGGB, a heady brew of math and magic

[austinpop]

9 minutes ago, muski said:

I have a 2020 iMac with an 8-core i7 with 64GB RAM and a very fast SSD (so swapfiles should work really well).

 

How much free space is on the SSD? For swapfiles to be created, there must be adequate free space on the SSD. If there's no free space left, then the OS can't grow the swap space, and hence will throw an Out of Memory error.

[muski]

5 hours ago, austinpop said:

 

How much free space is on the SSD? For swapfiles to be created, there must be adequate free space on the SSD. If there's no free space left, then the OS can't grow the swap space, and hence will throw an Out of Memory error.

I have 2.64 Terabytes of space available.

[muski]

6 hours ago, Zaphod Beeblebrox said:

Throttle DSD only helps with system cooling as it make sure not all cores are loaded. On Mac with no way to control swap, it is totally up to the OS. What is the max taps you have set? you can also email me the log so i can see what is going on. One way to help with swap is try and close all applications other than PGGB.  Is it a DSD64?

It's DSD64, The two tracks I'm having trouble with are both just over 16 minutes, so >600Mb as input files.

 

I've closed all other apps and have max taps on Auto. I'll email my log files.

 

cheers,

muski

[ted_b]

I am please to announce that, starting Aug 6, NativeDSD will begin selling the Patricia Barber Clique album in 32/352.8k, the original recording format.  Zaphod, Rajiv and others asked us to do this and we obliged.  We'll see how the sales go.  I will push for other labels/albums as we move on.  👍

 

Note: also, we will sell the album in DSD1024 (the first stereo DSD1024 I am aware of) and 32 bit DXD multichannel

[austinpop]

1 hour ago, ted_b said:

I am please to announce that, starting Aug 6, NativeDSD will begin selling the Patricia Barber Clique album in 32/352.8k, the original recording format.  Zaphod, Rajiv and others asked us to do this and we obliged.  We'll see how the sales go.  I will push for other labels/albums as we move on.  👍

 

Note: also, we will sell the album in DSD1024 (the first stereo DSD1024 I am aware of) and 32 bit DXD multichannel

That’s great news Ted, especially for PGGB users. 32-bit DXD files retain more of the precision from the original master, giving more information for PGGB to work with.

 

I hope PGGBers in particular, and ASers in general, support and buy this format.

[Zaphod Beeblebrox]

2 hours ago, austinpop said:

I hope PGGBers in particular, and ASers in general, support and buy this format.

I believe it is not just PGGB, DACs that can accept 8FS or higher at 32bits without truncating to 24bits can potentially benifit from 32bit DXD for the simple reason that this is the original format and retains more of the information form the original signal without having undergone a re-quantization step (from 32 to 24 bits) which could  introduce noise and results in less information.

 

I am committed to buy  the Clique album (as a fan of Patricia Barber) when it comes out in Aug. When presented with an option of 24 bits and 32bits, I will certainly go for 32bits.

[Zaphod Beeblebrox]

18 hours ago, austinpop said:

 

How much free space is on the SSD? For swapfiles to be created, there must be adequate free space on the SSD. If there's no free space left, then the OS can't grow the swap space, and hence will throw an Out of Memory error.

It turns out it is a fast internal SSD drive and the only partition is the OS partition and it is a 4TB SSD with 2.6Tb of free space, so really bummed why it will not grow beyond about 120GB. Perhaps some mac gurus can chime in.

[austinpop]

4 minutes ago, Zaphod Beeblebrox said:

It turns out it is a fast internal SSD drive and the only partition is the OS partition and it is a 4TB SSD with 2.6Tb of free space, so really bummed why it will not grow beyond about 120GB. Perhaps some mac gurus can chime in.

Ugh. I know people love to hate on Windows, but at least on this issue, I’ll take the explicit allocation of paging space in Windows any day.

 

Don’t get me wrong, my laptop is an MBP, so I like MacOS too.

[Zaphod Beeblebrox]

2 minutes ago, austinpop said:

Ugh. I know people love to hate on Windows, but at least on this issue, I’ll take the explicit allocation of paging space in Windows any day.

 

Don’t get me wrong, my laptop is an MBP, so I like MacOS too.

Mac is mostly better, but for other days use VMWare Fusion or Parallels 🤣 

[happybob]

5 hours ago, austinpop said:

Ugh. I know people love to hate on Windows, but at least on this issue, I’ll take the explicit allocation of paging space in Windows any day.

 

Don’t get me wrong, my laptop is an MBP, so I like MacOS too.

I'm a new user of PGGB (have been testing and just bought my license). I love the resulting sound improvements! (Dave DAC with Sean Jacobs DC4, doing 16FS, 24 bits or 32bits being evaluated). But I am currently running it on a M1 MacBook Pro (>1TB of free SSD space and 16GB of RAM) so it's an underpowered machine for PGGB although I am wondering if you have any thoughts about:

 

1. Best way to optimize this Mac (regardless of speed of processing PGGB files) for higher taps (currently on classical pieces I'm limited to 32Mtaps unfortunately before the dreaded "out of application memory" error and then often a PGGB crash). Most "regular" albums with shorter tracks allow for 256Mtaps with no problems. Dang... 

 

2. Does a virtual windows machine on this M1 Mac (assuming lots of available SSD space for swap files) allow for much higher taps even on long tracks? Speed is not at all a priority, I don't care how long it takes to process the files within reason.

 

3. What might be a good and very very small Windows computer (laptop or very small not-a-tower computer). 

 

Thanks again austinpop for all of your help in the past on PGGB and other issues! 

[kennyb123]

7 hours ago, Zaphod Beeblebrox said:

It turns out it is a fast internal SSD drive and the only partition is the OS partition and it is a 4TB SSD with 2.6Tb of free space, so really bummed why it will not grow beyond about 120GB. Perhaps some mac gurus can chime in.

Memory pressure is what I think might cause the swap file size to grow.  I use iStat Menu to keep an eye on this.  If you see memory pressure pinned at 100% with the swap file not growing then maybe something is amiss.  If memory pressure stays below 100% then that points to macOS not wanting to allocate more swap file because it doesn’t think it needs it.

 

It does seem though that the macOS may be putting a cap on how much it allocates. I also have a MacBook Pro with 16 GB of RAM.  Let me fire up PGGB on it to see what happens.  The 120 GB limit may have something to do with Matlab not being native on an M1.

[austinpop]

3 hours ago, happybob said:

But I am currently running it on a M1 MacBook Pro (>1TB of free SSD space and 16GB of RAM)

 

3 hours ago, happybob said:

the dreaded "out of application memory" error and then often a PGGB crash

 

I'm not clear why MacOS, with >1TB of free space (presumably on you boot drive?) would cause the app to throw an OOM. Perhaps as @kennyb123 says, it could be an artifact of Intel emulation on M1.

 

Calling MacOS experts...

[ray-dude]

I have same issue on 2012 Mac Pro with 128GB (Intel) when processing some large DSD files.  The crash is the watchdog giving up the ghost.  I suspect the system is asking for swap but not getting it fast enough, so the watchdog barks.

 

If there was preallocated swap (like there is on Windows), we wouldn't be having this issue.

 

I do not have this issue when running Windows in VMware on the same Mac.  I preallocate the swapfile in windows, and Bob's your uncle

[kennyb123]

I just started gargle-blaming a DSD album with only 16 GB installed in my Mac Pro.  I'll report back later.

[happybob]

1 hour ago, ray-dude said:

I have same issue on 2012 Mac Pro with 128GB (Intel) when processing some large DSD files.  The crash is the watchdog giving up the ghost.  I suspect the system is asking for swap but not getting it fast enough, so the watchdog barks.

 

If there was preallocated swap (like there is on Windows), we wouldn't be having this issue.

 

I do not have this issue when running Windows in VMware on the same Mac.  I preallocate the swapfile in windows, and Bob's your uncle

Thanks for the perspectives on the Mac issues!  I'm thinking maybe the M1 Mac is the problem too. My current M1 MacBookPro has a single 2TB SSD that's still got 1TB of free space available. I have VMWare fusion but haven't run it yet on this M1 Mac. Maybe I do that, although I'd sure prefer just staying with the Mac directly and not having to run a VM. I do plan to upgrade to the "likely coming soon" M2 (or maybe M1x label) MacBook Pro that's rumored to have up to 64GB of RAM. But it's still a rumor although seems likely, but maybe not out for a few months. 

 

I'm going to do some more testing to see what's up, but for now I've been able to do 32Mtaps just fine for long tracks. Not near the desired number but a lot more than the 1M taps of my MScaler. And for normal (5min or so) tracks I can do 256Mtaps. 

 

I'll report back later after more testing. I'm still new to PGGB and maybe there's something more I can try that I haven't. Thanks!

[Zaphod Beeblebrox]

20 minutes ago, happybob said:

Thanks for the perspectives on the Mac issues!  I'm thinking maybe the M1 Mac is the problem too. My current M1 MacBookPro has a single 2TB SSD that's still got 1TB of free space available. I have VMWare fusion but haven't run it yet on this M1 Mac. Maybe I do that, although I'd sure prefer just staying with the Mac directly and not having to run a VM. I do plan to upgrade to the "likely coming soon" M2 (or maybe M1x label) MacBook Pro that's rumored to have up to 64GB of RAM. But it's still a rumor although seems likely, but maybe not out for a few months. 

 

I'm going to do some more testing to see what's up, but for now I've been able to do 32Mtaps just fine for long tracks. Not near the desired number but a lot more than the 1M taps of my MScaler. And for normal (5min or so) tracks I can do 256Mtaps. 

 

I'll report back later after more testing. I'm still new to PGGB and maybe there's something more I can try that I haven't. Thanks!

If you look at the logs (.log file), it will say how much RAM it is using for a track, if it says 16M of 16M, then you are most likely to run into a issue if your Mac wont create more swap. I wonder if you wait for a long time does it ever start processing?

[happybob]

34 minutes ago, Zaphod Beeblebrox said:

If you look at the logs (.log file), it will say how much RAM it is using for a track, if it says 16M of 16M, then you are most likely to run into a issue if your Mac wont create more swap. I wonder if you wait for a long time does it ever start processing?

Thanks ZB, I've been meaning to reply directly to you via email. I see the message:

 

Estimated memory per worker: 2GB of 16GB RAM

Sugested max workers: 8

 

I also frequently see the message "Error in function PGGB.Continue() at line 906."

Another thing - sometimes the process does restart but other times it doesn't restart ever. And sometimes the whole app crashes.  I've attached a failed upsampling attempt log...

 

PGGB_album_debug.log

 

So it does appear that the Mac is not able to do this particular track with anything more than 32Mtaps. Maybe an M1 Mac issue, maybe something else... 

 

[Zaphod Beeblebrox]

11 minutes ago, happybob said:

Thanks ZB, I've been meaning to reply directly to you via email. I see the message:

 

Estimated memory per worker: 2GB of 16GB RAM

Sugested max workers: 8

 

I also frequently see the message "Error in function PGGB.Continue() at line 906."

Another thing - sometimes the process does restart but other times it doesn't restart ever. And sometimes the whole app crashes.  I've attached a failed upsampling attempt log...

 

PGGB_album_debug.log 3.41 kB · 3 downloads

 

So it does appear that the Mac is not able to do this particular track with anything more than 32Mtaps. Maybe an M1 Mac issue, maybe something else... 

 

I think it is better to take the debugging offline. From the logs it looks like the UI was forcibly closed,  that is when you see the .continue() error. Also the logs indicate that even at 256M taps this file was close to being done, it shows it took about 400 seconds and max taps says 256M. Pease email me a link (dropbox or wetransfer) to the failed track and I will see if I  can replicate on my M1 Mac.

 

Another suggestion I have is drop your workers to 4 (click on the cocktail and change the number of workers to 4)

[kennyb123]

3 hours ago, kennyb123 said:

I just started gargle-blaming a DSD album with only 16 GB installed in my Mac Pro.  I'll report back later.

It's been chugging away slowly.  I must say that I've been impressed by the memory handling thus far.  Fingers crossed that it will continue just humming along.  I'll share some graphs tomorrow after the album completes processing.

[kennyb123]

PGGB was able to process an entire SACD's worth of music with only 16 MB of RAM in my Mac Pro.  I'm sure it wasn't an apples-to-apples comparison to the case where out-of-memory errors were occurring, but I am thinking that the issues there were due to the layer that allows Intel-native code to run on the M1.  

 

The maximum taps value capped all the tracks to 512 million taps.  This appears to have kept PGGB using a very reasonable amount of virtual memory.  The first chart shows the size of the paging file over the last 24 hours.  The second shows the memory pressure over the last 24 hours.  The swap file sufficiently relieved the memory pressure when needed, which is exactly what it should do.

 

Like I said, not apples to apples but most certainly this test was a test of PGGB running on an "underpowered Mac".  I swapped into my 2010 Mac Pro a 6-core tray with only 16 GB of RAM.  The tray I normally use is 12-core tray with 128 GB of RAM.  The M1 is more powerful for sure than the 6-core tray with only 16 GB.  I think when Matlab comes out with a native version on M1, the memory handling situation will improved there for sure.  

 

The graphs were produced using iStat Menus, which I highly recommend.  https://bjango.com/mac/istatmenus/

 

[ray-dude]

With my OOM issues, I was at 2B taps +.  When I manually capped off the number of cores, it ran cleanly.  Watching memory use tick up, it was pretty clear about when things tipped over.  All on Intel, so not related to a translation layer.

 

If you can dial down memory use (fewer taps, fewer cores) you can tune away from OOM zone.

 

 

[austinpop]

Very useful MacOS suggestions and observations, @ray-dude and @kennyb123.

 

"To OOM it may concern." 🙂

[happybob]

I love the "to OOM it may concern"! Ha! 

 

I'm doing some more testing (various numbers of taps and "workers") on my M1 Mac and sharing files with ZB for evaluation. No clear resolution yet although part of the problem may be the M1 Mac itself vs an older intel Mac. More news to come after further testing by me and further eval by ZB.

 

And I'll second @kennyb123 comments about istat menus!

[Progisus]

Further to my experiments with the signal path of mconnect for mpd (upnp) playback.

 

If I use mconnect on my ipad playing from mininserver on Antipodes to MPD player on Antipodes:

1) turning off wifi on the ipad eventually stops playback

2) turning wifi back on restarts playback

 

This makes me think that signal path is through the ipad and once buffered signal is used up the playback will stop. This is indicative of upnp/dlna play and not true MPD.

 

With true MPD the tie to the library is made through the mpd configuration. Once the controller (app) makes the connection, the file plays directly from the configured library to the player. If an album has been cued, the control app is no longer required.

 

Antipodes implementation of MPD is the direct link. One just needs to find a controller app that will communicate and read tags correctly. One that was working for me was Rigelian but it is having issues with the wave tags.

 

The sound from true mpd implementation is a step up from upnp/dlna but the hassle with tags has me back with roon/hqpe.

 

If anyone can confirm or correct me please do.

 

PGGB… It is the way.

[Fourlegs]

31 minutes ago, Progisus said:

Further to my experiments with the signal path of mconnect for mpd (upnp) playback.

 

If I use mconnect on my ipad playing from mininserver on Antipodes to MPD player on Antipodes:

1) turning off wifi on the ipad eventually stops playback

2) turning wifi back on restarts playback

 

This makes me think that signal path is through the ipad and once buffered signal is used up the playback will stop. This is indicative of upnp/dlna play and not true MPD.

 

With true MPD the tie to the library is made through the mpd configuration. Once the controller (app) makes the connection, the file plays directly from the configured library to the player. If an album has been cued, the control app is no longer required.

 

Antipodes implementation of MPD is the direct link. One just needs to find a controller app that will communicate and read tags correctly. One that was working for me was Rigelian but it is having issues with the wave tags.

 

The sound from true mpd implementation is a step up from upnp/dlna but the hassle with tags has me back with roon/hqpe.

 

If anyone can confirm or correct me please do.

 

PGGB… It is the way.

 

Thanks for experimenting. Perhaps we Antipodes owners need to jointly put pressure on Antipodes (in a nice way) to put other players on their streamers that will support 705/768kHz files.

 

Innuos use Squeezelite on their streamers and they have found a way to get their 2.0 app with Squeezelite to be able to play 705/768kHz files. I will be asking Antipodes if they can do the same but please can others ask the same question! The squeaky wheel is the one they gets oiled!