DIY Budget PTP Grandmaster Clock with GPS correction

[El Guapo]

More precisely, I'm talking about how to build the boundary clock by very limited budget.

Ravenna/AES67 is heavily relying on PTPv2 (IEEE1588-2008). Although Zman-based Merging Anubis and Merging Hapi Mk II can be the PTP grandmaster clock in the Ravenna/AES67 network, but if look at the PTP clock source list, it indicates that the GPS has higher priority than internal oscillator. The Best Master Clock Algorithm, BMCA will always choose GPS-corrected one as the GMC.

But the price of GPS-corrected PTP GMCs are high. Fortunately we can build our own PTP GMC thru Linux system with reasonable expense: a retired PC with PCIe NICs and build as a PTP boundary clock. One NIC will be slave (to another GPS grandmaster clock) and all other NICs will be the master clock to the Ravenna network.

 

This was the Zman-based Merging Anubis as PTP GMC and the deviation showed on Hapi Mk II. It's about 7.6ppm @ DXD sample rate for me. The number actually was quite good, meets the AES67's requirement <±10ppm.

 

 

This was my home-made GPS PTP GMC. The deviation became <±1ppm. I think I can say it's an improvement.

 

The equipments I use:

- a current HQPlayer Embedded server running Ubuntu Server 22.04 / stock 5.15 low-latency kernel, still has two PCIe slots available for additional NICs.

- an affordable TM2000B GPS PTP GMC as clock source.

- a Merging customized NIC NET-MSC-GBEX1 (actually it's an Intel 82574L. I know it's very, very expensive compared to other 82574L but it worths. I'll talk about that later.)

- a retail, genuine Intel i350 dual-port NIC

Last three items was newly acquired and the total expense was about 1100USD.

 

Before setup we have to make sure the NICs support PTP hardware time stamping using this command:

ethtool -T <interface>

 

Here's Asus TUF Gaming's on-board Realtek RTL8125 2.5GbE NIC. Does not support hardware time stamping and no hardware clock. I noticed all of my Realtek NICs didn't support hardware time stamping. If you want to acquire Realtek NIC for PTP must keep in mind that it might just support software time stamping and does not suitable for the role of PTP GMC.

 

Here's the Merging's NET-MSC-GBEX1. It supports HW time stamping and a HW clock (numbered 0).

 

Here's the retail i350 and also looks good. Two HW clocks inside (numbered 1 and 2).

 

In Linux system we need to install LinuxPTP package:

sudo apt install linuxptp

 

LinuxPTP package contains ptp4l, phc2sys and pmc. The ptp4l is used for initialization, sanitize HW clocks on NICs and decides the role (MC, BC or TC). The phc2sys is used for synchronization between system clock and NIC's HW clock. The pmc is used for function check / verification.

 

The configuration for my low cost TM2000B is quite simple. I just power it on, put the antenna out of my window for better GPS satellite signal and change to two-step methods then leave there. Originally I tried to put TM2000B into Ravenna/AES67 network as GMC, but the GPS was too sensitive and completely destroy the audio stream--always stuttering for re-syncing the clock:

So I decided setup a boundary clock to make it more Ravenna/AES67 welcome.

 

I set Merging's NET-MSC-GBEX1 (enp9s0) as slave port. It has the best PTP HW clock which can gain 0ppm if use it solely in my environment (this is why I said it worths every penny).

Once the Merging's NET-MSC-GBEX1 synced with TM2000B it can be the best clock source after all. I believe Merging and Intel all did excellent job for choosing the quality of the HW clock for NET-MSC-GBEX1.

To configure i350 (enp5s0f0, enp5s0f1) as master port for Ravenna/AES67. I also adjusted Kp = 0.1  / Ki = 0.001 of proportional-integral servo to make smoother response. And also set PTP packets has highest priority (DSCP 46 this case).

 

Edit /etc/linuxptp/ptp4l.conf:

dscp_event		46
dscp_general	46
pi_proportional_scale		0.1
pi_integral_scale		0.001
boundary_clock_jbod		1
[enp9s0]
masterOnly		0
[enp5s0f0]
masterOnly		1
[enp5s0f1]
masterOnly		1

 

Back to CLI and enable / start the ptp4l service.

 

Next is to configure phc2sys to sync system clock. BC's mechanism would read and drop the time stamps from master (here is TM2000B), generate system clock's time stamp to master (here is i350), i350 reads and drops the system time stamp then use HW clock's time stamp to talk to Ravenna/AES67. Complex, huh? 🤣 So the system clock must be synced as well.

Just make a service (create /lib/systemd/system/phc2sys.service):

ExecStart=/usr/sbin/phc2sys -a -rr

This command can sync all clocks by master (here is Merging's NET-MSC-GBEX1). Enable / start the service.

 

After done these step, if you see the deviation is larger than Merging's internal clock, don't worry just completely shutdown your server to reset the HW clock. Start the server and check the ptp4l status again:

sudo pmc 'get PARENT_DATA_SET' 'get CURRENT_DATA_SET' 'get PORT_DATA_SET' 'get TIME_STATUS_NP' -u -b 0

 

Here's the status of my system. The GMC's MAC address F7-DA-B9 was from Merging's NET-MSC-GBEX1. The ***f7dab9-2 had a FAULTY code because I didn't connect the second port of i350 to any network so it's normal. Other than that everything looks quite good.

 

And the statistics in Hapi's Advanced page also looks good. The network delta drives by Kp = 0.1 / Ki = 0.001 PI servo. You can try other numbers but I still suggest 0.1 / 0.001 for the best result.

 

[Mantheunknown]

Hi Guapo,

 

Please help me understand the end result of this MC inclusion.  Does it improve the SQ of your Merging Hapi & Anubis significantly.  I realize that improvement is always the end goal.  Can a Mutec REF 20 (it is not GPS) MC provide a WC input (via some WC generator) to achieve something similar/close to it.  Curious as I have some of the components but not the intellect or energy to follow your trail blaze.

Do advise.  Cheers!

[El Guapo]

Hi the unknown

1 hour ago, Mantheunknown said:

Does it improve the SQ of your Merging Hapi & Anubis significantly.

Difficult to distinguish. Only the ppm number makes me feel good. 🤣 Maybe it's the placebo effect but if really want to say something... it has better clarity @ 48KHz sampling rate compared to Merging's Zman.

 

1 hour ago, Mantheunknown said:

Can a Mutec REF 20 (it is not GPS) MC provide a WC input (via some WC generator) to achieve something similar/close to it.

I can't find the spec sheet of Mutec Ref 20... based on current product of Mutec you can try MC3+ as word clock generator and connect it to Hapi's BNC WC input. Hapi only accepts WCK, no 10MHz. Then set Hapi as PTP GMC (remember to select the WCK as the clock source) and put it to crown position in the Ravenna/AES67 network. I think you can get very good result from it.

 

Guapo

[Mantheunknown]

Hi El Guapo,

 

I was referring to Ref 10 SE 120 which provides 10 Mhz clock output to feed a WCK generator.  I kind of swallowed some alphabets and numbers in between!!  I have a Singxer SU2 which can take 10MHz clock from the Mutec and output a word clock of 44.1K-384K.  So I could experiment to see effect on SQ. Thanks  Cheers

[Erik Haas]

@El Guapo  pretty interesting stuff..    I actually started looking into adding an external clock to my Hapi now that I have things integrated correctly but finding this is an entirely new rabbit hole!  lol

 

So how does a PTP Grandmaster clock differ from adding an external word clock ( with or without the addition of a master clock signal to the word clock)

 

Since Hapi is running off PTP does that mean adding a wordclock is pointless?

[El Guapo]

4 hours ago, Erik Haas said:

how does a PTP Grandmaster clock differ from adding an external word clock ( with or without the addition of a master clock signal to the word clock)

If you have a medium to large scale of AoIP network, cross-country subnet or multiple sampling rate (multiple PTP domain) in your network, the PTP GPS GMC is the best bet. All nodes will be synced by single time source: GPS. Although WC-synced Hapi Mk2 can be the GMC in small scale AoIP network but the WC is still running it's own clock and wandering over time (not synced to GPS or other ref signal). Also most of the WCs only support up to 192KHz, no DXD/DSD. PTP GMC has no such issue.

Some of the affordable GPS NTP time servers have 10MHz ref signal output. If your WC can accept 10MHz ref signal you may use the NTP server as the

reference. The WC will be synced by GPS as well.

 

4 hours ago, Erik Haas said:

Since Hapi is running off PTP does that mean adding a wordclock is pointless?

Some point of view... yes. Only one time source when operating the AoIP. When selecting PTP as Hapi's time source, the WC will be in STBY mode.

[Erik Haas]

On 7/6/2023 at 8:39 PM, El Guapo said:

If you have a medium to large scale of AoIP network, cross-country subnet or multiple sampling rate (multiple PTP domain) in your network, the PTP GPS GMC is the best bet. All nodes will be synced by single time source: GPS. Although WC-synced Hapi Mk2 can be the GMC in small scale AoIP network but the WC is still running it's own clock and wandering over time (not synced to GPS or other ref signal). Also most of the WCs only support up to 192KHz, no DXD/DSD. PTP GMC has no such issue.

Some of the affordable GPS NTP time servers have 10MHz ref signal output. If your WC can accept 10MHz ref signal you may use the NTP server as the

reference. The WC will be synced by GPS as well.

 

Some point of view... yes. Only one time source when operating the AoIP. When selecting PTP as Hapi's time source, the WC will be in STBY mode.

Interesting.   Though if using HQplayer and if Hapi is used as a master clock can the PTP clock add better clocking / audio quality to the Hapi?

 

Then I also question, if youre running a separate PTP grand master clock, wouldnt you need to add a switch that is also PTP capitable to distribute the master lock signal to everything?

 

I did some searching and it looks like Sonifex makes a pretty nice GM PTP clock which is also Ravenna certified.

 

https://www.sonifex.co.uk/avn/avn-gmcs.shtml

 

 

[El Guapo]

11 hours ago, Erik Haas said:

if using HQplayer and if Hapi is used as a master clock can the PTP clock add better clocking / audio quality to the Hapi?

In this scenario, if Hapi mk2 is the only output for HQPlayer, add a PTP GMC might have not much help (zman-based system already has a very high quality clock inside). Hapi still can connect a word clock as clock source tho.

 

11 hours ago, Erik Haas said:

if youre running a separate PTP grand master clock, wouldnt you need to add a switch that is also PTP capitable to distribute the master lock signal to everything?

If 3 or 4 AES67 equipments in the network and distance is short, non-PTP switch like Cisco CBS350 still doing great. Such switch is recommended by Merging. If equipments getting more and RJ45 cable distance getting longer, PTP-TC, or more expensive, PTP-OC capable switch is the better choice. Netgear M4250 is TC capable and recommended by Merging as well.

 

[Erik Haas]

On 7/12/2023 at 11:36 PM, El Guapo said:

In this scenario, if Hapi mk2 is the only output for HQPlayer, add a PTP GMC might have not much help (zman-based system already has a very high quality clock inside). Hapi still can connect a word clock as clock source tho.

 

If 3 or 4 AES67 equipments in the network and distance is short, non-PTP switch like Cisco CBS350 still doing great. Such switch is recommended by Merging. If equipments getting more and RJ45 cable distance getting longer, PTP-TC, or more expensive, PTP-OC capable switch is the better choice. Netgear M4250 is TC capable and recommended by Merging as well.

 

All Great info...   I dove down the rabbit hole for a few weeks looking at stuff.   Still not sure if it's advisable to look into PTP clocking or finding a Word Clock generator capable  of doing fs x256 rates which i would assume would take care of 11.2mhz 256dsd rates.

 

I remember reading that it's important to have a low phase / jitter clock when using clocking for audio.   Does PTP clocking have the same low phase / jitter characteristics? 

[El Guapo]

33 minutes ago, Erik Haas said:

finding a Word Clock generator capable  of doing fs x256 rates

Although some word clock supports 11.2MHz for DSD256 but I'm afraid it can't be used on Hapi. 🤔️ If you have DSD requirement I'd suggest to find a PTP GMC.

 

33 minutes ago, Erik Haas said:

Does PTP clocking have the same low phase / jitter characteristics?

The PTP uses IP packets carried with time stamp to help clients to find the correct time difference, then use such info to synchronize its time to the GMC. There's no ref clock signal transmitted in the network so don't worry about the phase / jitter specs. AFAIK most of the GPS PTP GMCs use 1 PPS from GPS chip to calibrate its own clock (including frequency, phase and ToD), then use that calibrated clock as the source for PTP. I think PTP GPS GMC for Ravenna all have good performance (GPS accuracy can count to sub-microsecond). 

 

BTW I found an error in my previous reply. It's "PTP-BC capable", not OC. BC = Boundary Clock.

[Mantheunknown]

Hello El Guapo,

 

Please help me understand why word clock supporting 11.2 MHz for DSD 256 cannot be used on HAPI.  Where/what is the limitation.

Thanks.  Cheers

 

[El Guapo]

14 minutes ago, Mantheunknown said:

why word clock supporting 11.2 MHz for DSD 256 cannot be used on HAPI

AFAIK Merging's DSD256 uses 352.8KHz, not 11.2MHz. I also checked Merging+CLOCK spec it does not have 11.2MHz. So I concluded 11.2MHz might not work on Hapi with *P boards. Maybe I was wrong... 🤔️

[El Guapo]

13 hours ago, Erik Haas said:

it's important to have a low phase / jitter clock when using clocking for audio

Screenshot is about how PTP works... IP *.104 is my Intel e810 (as PTP GMC).

 

In default the AES67 media profile the GMC will announce (sounds like: hey dudes, I'm going to tell you the time) every 1 second, and send sync message every 0.125 seconds for syncing. The follow up messages are by two-step configuration which contain the actual time stamps. After followers receive the sync message it'll send delay request to the GMC. GMC then responses the delay. After such mechanism the followers can find the time difference between GMC and the followers. Followers then use those data to sync the local clock.

 

Here's the actual packets I captured from my Ravenna network (I added a magenta arrow to represent time sequence).

 

Although there's no jitter or phase issues in PTP time sync mechanism but the delay issue from the switch is significant. Screenshot shows the packets had been delayed for nearly 4us. So if your Ravenna network is complex like multichannel, hi-res contents or more equipments, I'd suggest to configure a TC-capable switch at least. 

 

[El Guapo]

2 hours ago, El Guapo said:

announce every 1 second

Correction again... announce every 0.5 second.😅

[Erik Haas]

@El Guapo thank you again for all this info and your patience with my questions.

 

I have found myself diving deep in the PTP rabbit hole for the past few weeks and “think” I have a better understanding.  
 

please correct me if my understanding is incorrect.  

To improve the clock accuracy of a unit like the Hapi/Anubis via PTP the following are needed 

 

1. PTP grandmaster clock.  This may use GPS as the highest clock priority and May have a internal clock for back up

 

2. PTP Boundary Clock / Switch : this goes in between the GMC and your Slave Devices (Hapi/Anubis) and distributes the time corrected GMC.  ( it basically measures and corrects the time stamp delays)

 

There  has been some recent  confusion with Switches such as the Netgear M4250 being used to improve clocking. From my understanding that this device is a transparent clock meaning it only distributes the GMC signal to other boundary clocks and has no real effect on overall PTP accuracy in our use case.  
 

Outside of building what you did it seems like the following devices would be needed to implement a highly accurate PTP clocking system would require the following as an example:

 

1.  Grandmaster Clock : Sonifex AVN-GMCS

2. Boundary Clock/Switch : Quarra PTP - 1G Compact

3. Merging Hapi/Anubis 

 

 

thank for letting think this out on the thread.  Hopefully my understanding is correct.  

 

[El Guapo]

10 hours ago, Erik Haas said:

2. PTP Boundary Clock / Switch : this goes in between the GMC and your Slave Devices (Hapi/Anubis) and distributes the time corrected GMC.  ( it basically measures and corrects the time stamp delays)

Boundary clock is more like an agent / middleman. BC will be calibrated by GMC then acts as a GMC. Clients won't talk to the real GMC after all.

GMC --> BC <-> clients structure is to reduce the GMC's workload (imaging hundreds of clients send delay request at the same time in large scale network...) and also be the backup GMC if the real GMC failed. 

 

10 hours ago, Erik Haas said:

There  has been some recent  confusion with Switches such as the Netgear M4250 being used to improve clocking. From my understanding that this device is a transparent clock meaning it only distributes the GMC signal to other boundary clocks and has no real effect on overall PTP accuracy in our use case.

TC-capable switch will update the time stamp when packets leave for letting the next client knows how many delay in the switch. So it still affects overall PTP accuracy.

GMC <-> TC <-> clients structure still can gain very high accuracy (nanosecond level) because clients can talk to the GMC directly. In small scale Ravenna network it is quite sufficient. You can even manually increase the announce / sync intervals for more accurate time syncing process if you like (in such case only few clients in the network so it won't cause high workload for GMC).

 

10 hours ago, Erik Haas said:

Outside of building what you did it seems like the following devices would be needed to implement a highly accurate PTP clocking system would require the following as an example:

 

1.  Grandmaster Clock : Sonifex AVN-GMCS

2. Boundary Clock/Switch : Quarra PTP - 1G Compact

3. Merging Hapi/Anubis

If your budget is ok this list is the best equipments you can have so far.🤤