Optical Network Configurations

[jabbr]

36 minutes ago, R1200CL said:

We definitely shouldn’t need a central atomic clock, but maybe some technology that the White Rabbit Project is implementing in switches can be adopted. ... And why is CERN developing almost jitter free switches? (Why aren’t they endorsing 400GB ?). 

 

To be clear about this, not about jitter on network interface, rather very precise time synchronization over very large distances. Do you know that the LHC physical plant at CERN spans from France to Geneva? I can envision situations where they may need to time sync the experiment very closely over km. 

[plissken]

7 hours ago, jabbr said:

Now that the HST market has moved beyond 10Gbe, these Solarflare NICs can be picked up very cheap on eBay.

I'm running SolarFlare. The drivers are rock solid and their software suite (including remote, over the network firmware updates) is terrific.

[lmitche]

18 hours ago, jabbr said:

 

To be clear about this, not about jitter on network interface, rather very precise time synchronization over very large distances. Do you know that the LHC physical plant at CERN spans from France to Geneva? I can envision situations where they may need to time sync the experiment very closely over km. 

LOL, the north west edge of the Geneva airport runway defines the Swiss border with France. So France to Geneva isn't a huge distance by any means, especially in fiber terms.

[jabbr]

4 hours ago, lmitche said:

LOL, the north west edge of the Geneva airport runway defines the Swiss border with France. So France to Geneva isn't a huge distance by any means, especially in fiber terms.

 

Its not huge in terms of fiber terms, obviously fiber encircles the globe, rather:https://home.cern/science/accelerators/large-hadron-collider https://home.cern/resources/faqs/facts-and-figures-about-lhc what makes it "very large" is in terms of a scientific experiment which requires high precision. You might imagine that they have timing needs which go beyond home audio stereo synchronization  

[R1200CL]

4 hours ago, jabbr said:

 

Its not huge in terms of fiber terms, obviously fiber encircles the globe, rather:https://home.cern/science/accelerators/large-hadron-collider https://home.cern/resources/faqs/facts-and-figures-about-lhc what makes it "very large" is in terms of a scientific experiment which requires high precision. You might imagine that they have timing needs which go beyond home audio stereo synchronization  

Well, they didn’t go for 10GB or above in the White Rabbit project 😀

 

Remember my point was if any of that technology used in their switches, could be adopted in “audio switches”.
Not the time synchronization in long distances, which I think those guys are after.

 

(Well unless we’re starting our DAC to synchronize with Tidal, Qobuz etc. Wow, Roon in cloud with atom clocks syncing to your DAC 🤓).

 

Actually important in 5G network. Look at Cisco solutions if someone like to study more.

 

So far no one did brother to comment on different jitter specs in SFP data sheets. I don’t have the skills to interpret everything those sheets says. But, 0,0007 compared to 23 seems to make a huge difference. So I was hoping you or @plisskencould enlighten us. If you guys don’t know either, maybe you know who to contact. 
 

And why did they (WR) even bother to develop switches in 1GB, if 10 GB and above should be sufficient when it comes to jitter. 
I would expect these guys know what they’re talking about. 
 

Also the standard mentioned is more than 600 pages, meant to cover everything. Those who “hide” behind that or other standard. Come and show me how that standard demands no jitter. When I scanned through those 600 pages, I didn’t see any eye pattern. I find manufacturers with expensive equipment showing off pictures so measurements can be made. 
 

Again why isn’t there eye pattern available for SPF modules and switches. They do go up to almost 5GB. 

 

And if you study the Finsair selection chart, there is only one module (FTLX1471D3BCV which they won’t tell jitter. Only comply with a spec from 2005), that will comply with optical Ethernet. Rest will only work with 10GB as I understand it. 
So unless you’re building 10GB all over, you will probably struggle, unless you’re using a PC as endpoint. (No other device will comply). 

Also, I’m still not sure it the jitter reduction is related to switch vs module or a combination. 
Does the switches display jitter info in data sheets ?

 

EDIT:

The 10 Gigabit Ethernet (10GbE) specifications are defined in clauses 44 through 54 of the IEEE 802.3-2005 standard. It contains critical timing specifications governing electrical implementations such as 10GBASE-X4 as well as optical interfaces such as the 10GBASE-R. Although the standard specifies jitter limits for the transmitter and the receivers, it does not impose explicit restrictions on the interface reference clock.

 

 

[The Computer Audiophile]

24 minutes ago, R1200CL said:

Well, they didn’t go for 10GB or above in the White Rabbit project 😀

 

Remember my point was if any of that technology used in their switches, could be adopted in “audio switches”.
Not the time synchronization in long distances, which I think those guys are after.

 

(Well unless we’re starting our DAC to synchronize with Tidal, Qobuz etc. Wow, Roon in cloud with atom clocks syncing to your DAC 🤓).

 

Actually important in 5G network. Look at Cisco solutions if someone like to study more.

 

So far no one did brother to comment on different jitter specs in SFP data sheets. I don’t have the skills to interpret everything those sheets says. But, 0,0007 compared to 23 seems to make a huge difference. So I was hoping you or @plisskencould enlighten us. If you guys don’t know either, maybe you know who to contact. 
 

And why did they (WR) even bother to develop switches in 1GB, if 10 GB and above should be sufficient when it comes to jitter. 
I would expect these guys know what they’re talking about. 
 

Also the standard mentioned is more than 600 pages, meant to cover everything. Those who “hide” behind that or other standard. Come and show me how that standard demands no jitter. When I scanned through those 600 pages, I didn’t see any eye pattern. I find manufacturers with expensive equipment showing off pictures so measurements can be made. 
 

Again why isn’t there eye pattern available for SPF modules and switches. They do go up to almost 5GB. 

 

And if you study the Finsair selection chart, there is only one module (FTLX1471D3BCV which they won’t tell jitter. Only comply with a spec from 2005), that will comply with optical Ethernet. Rest will only work with 10GB as I understand it. 
So unless you’re building 10GB all over, you will probably struggle, unless you’re using a PC as endpoint. (No other device will comply). 

Also, I’m still not sure it the jitter reduction is related to switch vs module or a combination. 
Does the switches display jitter info in data sheets ?

 

 

 

 

I’m talking to a vendor now about WR switches :~)

[jabbr]

1 minute ago, The Computer Audiophile said:

I’m talking to a vendor now about WR switches :~)

Great, there is nothing preventing a 1Gbe switch from having defined jitter/eye pattern specifications, nor adapting the "stressed eye pattern" testing methodology to assure jitter rejection.

[jabbr]

29 minutes ago, R1200CL said:

Remember my point was if any of that technology used in their switches, could be adopted in “audio switches”.
Not the time synchronization in long distances, which I think those guys are after.

 

(Well unless we’re starting our DAC to synchronize with Tidal, Qobuz etc. Wow, Roon in cloud with atom clocks syncing to your DAC 🤓).

 

Actually important in 5G network. Look at Cisco solutions if someone like to study more.

 

So far no one did brother to comment on different jitter specs in SFP data sheets. I don’t have the skills to interpret everything those sheets says. But, 0,0007 compared to 23 seems to make a huge difference. So I was hoping you or @plisskencould enlighten us. If you guys don’t know either, maybe you know who to contact. 
 

And why did they (WR) even bother to develop switches in 1GB, if 10 GB and above should be sufficient when it comes to jitter. 
I would expect these guys know what they’re talking about. 

 

Sure, there is nothing preventing a 1Gbe switch vendor from optimizing certain aspects of the jitter rejection, or doing the stressed eye pattern testing to demonstrate jitter rejection. I don't have any awareness of their design decisions nor methodology.

 

I strongly suspect that the vast benefit of fiberoptic ethernet regarding audio relates to the absence of common mode noise transmission but that's just my speculation. I bring up 10Gbe ethernet, again, not because it is necessary but to point out that the standard involves compliance testing of jitter rejection. 

[jabbr]

If you want to delve into the nitty gritty details of the jitter rejection specifications, you can start here:

"Stressed receiver conformance test"  ... you are going to need to do some reading  ... needless to say these concepts were thought about extensively at and just after the turn of the century.

 

https://www.ieee802.org/3/axay/public/jan_08/dawe_2_0108.pdf

http://grouper.ieee.org/groups/802/3/10GEPON_study/email/pdf00050.pdf

[R1200CL]

Thanks @jabbr

 

From the first link, I noticed requirement is less than 0.25 UI peak-peak of jitter.

It also only applies to the SFP as I understand it. (Not the switch). 

 

It was helpful to find that number. 

IEEE 802.3-2005 standard:

https://d2cpnw0u24fjm4.cloudfront.net/wp-content/uploads/802.3-2005_section1.pdf

When it comes to require a stressed eye pattern test for the etherRegen, I think that will never happen, and it isn’t relevant. 

I’m even not sure if Alex and John moves up to 10GB, you will have such a test. 
Actually the opticalRendu is a switch inside, so your demand should include that device as well. 
 

However, if I’m right that most of the test and engineering done is much related to the SFP’s itself, I do agree very much that moving up to SFP+ and 10GB specifications ought to be done in future releases of streaming devices related to audio. 
I’m not sure if the audiophile market is ready for that step yet. 
With the price of that Mikrotik switch, it also may be challenging to upgrade the etherRegen to a 10GB version. (Which hasn’t been requested either, and probably is unnecessary).

 

By using the “correct” SFP’s in Lumin or opticalRendu, I’m also not sure how much better SQ can be achieved to upgrade to 10GB SPF+. 

So maybe we could discuss if the use of “correct” SFP modules is enough as of today. And moving up further won’t help us at all. 

 

[lmitche]

17 hours ago, jabbr said:

 

Its not huge in terms of fiber terms, obviously fiber encircles the globe, rather:https://home.cern/science/accelerators/large-hadron-collider https://home.cern/resources/faqs/facts-and-figures-about-lhc what makes it "very large" is in terms of a scientific experiment which requires high precision. You might imagine that they have timing needs which go beyond home audio stereo synchronization  

You are right of course.

 

Nevertheless your post made me think of how "thick or thin" an international border can be. You may be on to something.

[plissken]

14 hours ago, R1200CL said:

But, 0,0007 compared to 23 seems to make a huge difference. So I was hoping you or @plisskencould enlighten us.

 

1: I think there is a fundamental misunderstanding of 10G interface jitter vs audio clock jitter. They are wholly unrelated.

   If I transfer 1250MB 16/44.1 PCM track and buffer that on my end point all at once jitter mattered for 1 second.

   If I transfer 1250MB 16/44.1 PCM track and I buffer 1MB at a time, then jitter only mattered for 1/1250th of a second.

 

2: Ethernet, with no manipulation in place or contention, transfers at full wire right. Period. There's no are are partial rates. You will get the the highest speed of the lowest speed link in the chain (generally your ISP excepting some of you lucky bastards).

 

3: 0.0007 vs 23? What huge difference to you? To your audio?

[The Computer Audiophile]

23 minutes ago, plissken said:

 

1: I think there is a fundamental misunderstanding of 10G interface jitter vs audio clock jitter. They are wholly unrelated.

   If I transfer 1250MB 16/44.1 PCM track and buffer that on my end point all at once jitter mattered for 1 second.

   If I transfer 1250MB 16/44.1 PCM track and I buffer 1MB at a time, then jitter only mattered for 1/1250th of a second.

 

2: Ethernet, with no manipulation in place or contention, transfers at full wire right. Period. There's no are are partial rates. You will get the the highest speed of the lowest speed link in the chain (generally your ISP excepting some of you lucky bastards).

 

3: 0.0007 vs 23? What huge difference to you? To your audio?

These numbers are all theoretical. With audio devices one can't pick and chose what to buffer or how large/small the buffer is in the endpoint. Most manufacturers won't tell you the size of the buffer either. 

 

The speed at which an audio endpoint can handle data is far less than wire speed, even if rated for say 1G. These are usually low power ARM devices. 

 

It's all very interesting to me and I'd love to dig in further, with you of course because I value your expertise. 

 

P.S. I'm one of those lucky bastards with 1G up/down from my ISP for $65 per month. I've been thinking about bringing in a second strand of fiber or even 1G service from another ISP for failover. Yes, I know, first world problems :~)

[jabbr]

3 hours ago, R1200CL said:

Thanks @jabbr

 

From the first link, I noticed requirement is less than 0.25 UI peak-peak of jitter.

It also only applies to the SFP as I understand it. (Not the switch). 

This is *everything* in the link — end to end. That’s the point. When you plug it all together, it has to work. The clock, the board layout, the power supplies, the SFP module. Switch, NIC whatever — all the jitters of everything added up have to be under the limit. If upstream jitter (the stress) made it through it would blow the timing.

[R1200CL]

3 hours ago, plissken said:

What huge difference to you? To your audio?

Very good question. 
At least I can be confident the SFP is within limitations with a good margin. 

 

What makes this a bit confusing is other modules from Finisar use different parameters. 

 

To my audio, probably not much. But at least I hope I will have a jitter free transfer 😀

However people do report differences in SFP’s, so that’s why I found best possible SFP and have just purchased them. So I will know in two to four weeks. 

Not everyone can control buffers and do things your way. 
 

BTW: I think your famous $8000 test, has nothing to do with testing switches and their jitter. 
But I can see you have a good point. Which also raises some questions about how streamers and endpoint ought to be designed. 
 

Here is an interesting link about White Rabbit switches and phase noise, jitter. 
https://afi-project.jinr.ru/attachments/download/119/mgr.pdf

 

 

 

[plissken]

8 hours ago, The Computer Audiophile said:

With audio devices one can't pick and chose what to buffer or how large/small the buffer is in the endpoint. Most manufacturers won't tell you the size of the buffer either. 

 

That depends though. I went the scalable bricks route vs all in one. So for me I have 1GB of buffer.  I have 332MB/s over 10GB fiber.

 

I think asking a manufacturer how they buffer and how much they buffer are good questions.

[plissken]

8 hours ago, The Computer Audiophile said:

The speed at which an audio endpoint can handle data is far less than wire speed, even if rated for say 1G. These are usually low power ARM devices. 

 

Given the cost of some of the streamers an embedded i86 solution would be in order. I think a $4000 should have solid 1G performance.

[R1200CL]

.......well, I guess pulling the network plug should give an indication 😀

[jabbr]

10 hours ago, The Computer Audiophile said:

The speed at which an audio endpoint can handle data is far less than wire speed, even if rated for say 1G. These are usually low power ARM devices. 

The low power ARM devices are getting better and better. Nvidia is owning the ARM compute on high speed NIC space ... but the new RaspPi’s are solid 1Gbe.

[R1200CL]

7 hours ago, jabbr said:

This is *everything* in the link — end to end. That’s the point. When you plug it all together, it has to work. The clock, the board layout, the power supplies, the SFP module. Switch, NIC whatever — all the jitters of everything added up have to be under the limit. If upstream jitter (the stress) made it through it would blow the timing.

Well, your conclusion seems reasonable, but it may be a bit more complicated than that. 
 

We are discussing this based on a wish of best SQ, obtained by lowest jitter, phase noise, and whatever more is mentioned in John’s white paper. (Well, at least I am). 

The 10GB transport system is much more. 
Here is some information and different test. Stressed eye test applies only to the optical part is my understanding. 
https://www.design-reuse.com/articles/5404/testing-10-gigabit-ethernet.html

 

Also if to follow your conclusion, we would have to purchase a system from our router to our DAC guaranteed from one single manufacturer. Most of us will purchase parts separately, from different manufacturers. 

Different equipment require different test, I would say. So far I’ve only focused on 1 GB network, as that’s the only interface available towards endpoint, DAC’s and streamers, unless your building your own PC. (Which may introduce other problems again). 
 

It seems there’s some good engineering hidden in SFP design which we are after. 
I don’t know if parameters in managed switches will help us in any way, like QoS or similar. 
However it’s (or should) already beyond any doubt, be accepted that switches matters in certain (most?) setups. The way @plissken is doing this, a switch won’t matter. 
A 1GB stick would be equal in his case I think 😀(His test only proves jitter won’t exist if buffers is in use). 

Have a look at this white paper. https://support-kb.spirent.com/resources/sites/SPIRENT/content/live/FAQS/10000/FAQ10597/en_US/How_to_Test_10G_Ethernet_WhitePaper_RevB.PDF

Does it actually have much relevance for what we’re after ?

 

Isn’t John’s white paper then more relevant.

There is some interesting test that may be carried out. 
Take the Mikrotik switch add, a RJ45 module towards an ultraRendu. (Or microRendu). Then test  it against etherRegen inserted. Any difference between etherRegen in the chain or not. We may also use a 2960G as well for a third setup. With switches in the chain, you would add SFP’s as well. (One SFP+). 

 

Add a FMC (opticalModule) in the chain and test. Does 10GB FMC exist BTW ?
 

Use power supplies like LPS 1.2 and/or Paul Hynes SR4T. 
 

Exchange the ultraRendu with an opticalRendu and do test. Inserting an etherRegen seems probably a bit odd, but anyway it interesting to know if any difference. 
 

Do same test with Lumin. Should be very interesting if you can alter the inputs to Lumin on the fly. 
(Hm, have Lumin huge buffer ?)

 

 

 

[jabbr]

11 hours ago, R1200CL said:

I noticed requirement is less than 0.25 UI peak-peak of jitter.

To make another explanation point, consider that 10Gbe UI is 1/10 that of 1Gbe and 100 Gbe 1/100 of 1Gbe, so the allowable amount of jitter goes down by an order of magnitude or two. That means that *everything* needs to get that much better.

 

Ok now realize that the 100 Gbe and 200 Gbe NICs which run on the PCIe bus are powered off the backplane with bock standard power supplies. They still meet these insane standards! The engineering required is mind boggling!

[R1200CL]

9 minutes ago, jabbr said:

To make another explanation point, consider that 10Gbe UI is 1/10 that of 1Gbe and 100 Gbe 1/100 of 1Gbe, so the allowable amount of jitter goes down by an order of magnitude or two. That means that *everything* needs to get that much better.

 

Ok now realize that the 100 Gbe and 200 Gbe NICs which run on the PCIe bus are powered off the backplane with bock standard power supplies. They still meet these insane standards! The engineering required is mind boggling!

Is Ethernet a protocol or transfer technology relevant when we go above 10GB ?

Im asking course there is almost no modules in SPF+ and above that supports 1 GB Ethernet that we must have. (Unless you says we make the discussion only relevant for computer endpoint). 

[jabbr]

14 minutes ago, R1200CL said:

Isn’t John’s white paper then more relevant.

Um, I think what is important is that you like the SQ and I have heard many great things about the SQ of the opticalRendu and there are many people who love the EtherRegen with copper endpoints. 
 

That’s what’s important. I’m not calling for testing beyond what 1Gbe calls for.

 

The “white paper” is marketing. If someone thinks that there is jitter transmitted through a switch, then show me some stressed receiver measurements done with a 10 Gbe switch. I am asserting this doesn’t happen with compliant equipment. 

[R1200CL]

5 minutes ago, jabbr said:

If someone thinks that there is jitter transmitted through a switch

Eh.., Just to be sure:

Do we agree on switches can transfer jitter/ phase noise etc ?

[jabbr]

19 minutes ago, R1200CL said:

Eh.., Just to be sure:

Do we agree on switches can transfer jitter/ phase noise etc ?

The jitter which is possibly transmitted through a 10Gbe switch is standardized to be negligible (you need to pass the stressed receiver test which injects jitter) So no. 
 

Furthermore no 1Gbe switch vendor has demonstrated better than 0.025 UI jitter when jitter has been injected. 
 

And let me reiterate. I think the products are great. If there is equipment on the network which has jitter, it’s likely cheap 1Gbe or slower equipment.