Optical Network Configurations

[Bertel]

Oh wow - to further add to the confusion (I hope not) and to add another data point regarding SFP28 compatibility:

 

I just remembered that there is not only QSFP28 (and up) which builds onto the different layout and architecture of QSFP, but also SFP28 as highest rate for the earlier and smaller SFP layout and architecture. So there is / should be downward compatibility of SFP28 (25G) -> SFP+ (10G) -> SFP (1G).

 

I still had a couple of Finisar FTLF1436P3BCL modules lying around (which I had bought previously in an attempt to scale up my fibre network from 10G to 25G, but then wasn't able to fing a 25G switch which was silent enough to run it in my flat, so the project was abandoned). These modules are SFP28, and their data sheet says that they support 'up to 25.78 Gb/s bi-directional data links'. Mind here: no specific rates mentioned as in the data sheets of other Finisar modules I have, but rather a range of 'up to', which to me was an indicator that maybe also 10G and even 1G could work.

 

So I just replaced all my modules:

• In the MikroTik CRS305 switch:
  • two Finisar FTLX1435D3BTL for my 10G connections to my music server with HQPlayer and to my Mikrotik RB4011iGS+5HacQ2HnD (oh these Mikrotik names... It is my fibre to wireless 'converter' to connect to and isolate from my internet router)
  • one Finisar FTLX1435D3BCV dual rate module for the 1G connection to my Signature Rendu
• In the Signature Rendu: another Finisar FTLX1435D3BCV as corresponding endpoint for the 1G connection
• In my music server in the Intel X520-2 network adapter
• (I only had 5, so need to buy another one for the Mikrotik TB4011)

And guess what: all worked without any issue, the connections just continued, and still do so after several reboots of all devices to make sure. This time the replacements were pure plug and play.

 

Will still monitor of course, to ensure all continues to go right. Yet along the lines of our previous thinking, i.e. that for reasons of best jitter performance 10G modules (single or dual rate) are superior to 1G modules and therefore should be used preferrably, I'd assume that SFP28 modules with their even higher requirements for jitter performance (I think @jabbr you had mentioned that a while back) should even be 'more better' ;-) I don't know about other parameters that might eat up the benefits, like some sort of noise increase due to higher bandwidth or energy consumption, the higher temperature of these SFP28 modules compared to the SFP+ ones (10C more), or what seems to be different Bias and Power levels (see screenshot of SwOS 'SFP' tab in CRS305)...

 

So for the moment I'm using 25G SFP28 modules for my 10G and 1G connections. Weird, but I hope theory is on my side and that's the best possible fibre network performance I can get.

 

[Johnnydev]

1 hour ago, Bertel said:

Oh wow - to further add to the confusion (I hope not) and to add another data point regarding SFP28 compatibility:

 

I just remembered that there is not only QSFP28 (and up) which builds onto the different layout and architecture of QSFP, but also SFP28 as highest rate for the earlier and smaller SFP layout and architecture. So there is / should be downward compatibility of SFP28 (25G) -> SFP+ (10G) -> SFP (1G).

 

I still had a couple of Finisar FTLF1436P3BCL modules lying around (which I had bought previously in an attempt to scale up my fibre network from 10G to 25G, but then wasn't able to fing a 25G switch which was silent enough to run it in my flat, so the project was abandoned). These modules are SFP28, and their data sheet says that they support 'up to 25.78 Gb/s bi-directional data links'. Mind here: no specific rates mentioned as in the data sheets of other Finisar modules I have, but rather a range of 'up to', which to me was an indicator that maybe also 10G and even 1G could work.

 

So I just replaced all my modules:

• In the MikroTik CRS305 switch:
  • two Finisar FTLX1435D3BTL for my 10G connections to my music server with HQPlayer and to my Mikrotik RB4011iGS+5HacQ2HnD (oh these Mikrotik names... It is my fibre to wireless 'converter' to connect to and isolate from my internet router)
  • one Finisar FTLX1435D3BCV dual rate module for the 1G connection to my Signature Rendu
• In the Signature Rendu: another Finisar FTLX1435D3BCV as corresponding endpoint for the 1G connection
• In my music server in the Intel X520-2 network adapter
• (I only had 5, so need to buy another one for the Mikrotik TB4011)

And guess what: all worked without any issue, the connections just continued, and still do so after several reboots of all devices to make sure. This time the replacements were pure plug and play.

 

Will still monitor of course, to ensure all continues to go right. Yet along the lines of our previous thinking, i.e. that for reasons of best jitter performance 10G modules (single or dual rate) are superior to 1G modules and therefore should be used preferrably, I'd assume that SFP28 modules with their even higher requirements for jitter performance (I think @jabbr you had mentioned that a while back) should even be 'more better' ;-) I don't know about other parameters that might eat up the benefits, like some sort of noise increase due to higher bandwidth or energy consumption, the higher temperature of these SFP28 modules compared to the SFP+ ones (10C more), or what seems to be different Bias and Power levels (see screenshot of SwOS 'SFP' tab in CRS305)...

 

So for the moment I'm using 25G SFP28 modules for my 10G and 1G connections. Weird, but I hope theory is on my side and that's the best possible fibre network performance I can get.

 

Does it sounds better? 

[Bertel]

59 minutes ago, Johnnydev said:

Does it sounds better? 

 

I can't tell a difference, but I don't listen to / compare network components anyway. It sure sounded awesome before and sounds awesome now 😉

[dbastin]

1 hour ago, Bertel said:

 

I can't tell a difference, but I don't listen to / compare network components anyway. It sure sounded awesome before and sounds awesome now 😉

Well, that is a lot better than people who can't theirs to working get ... silence.

[Jud]

9 hours ago, Jud said:

Edit: I've sent a note to C-Lab asking whether SFP+ modules at both ends of the Fitlet3-QNAP connection would work. Let's see what they say.

Got a nice note back promptly that says the optical input for Fitlet3 will do 1G only. Doesn't sound like they have any plans (or even possibility?) for a 10G upgrade. So I'll just use a couple of spare 1G modules I have around on either end of that connection.

[jabbr]

10 hours ago, Bertel said:

I still had a couple of Finisar FTLF1436P3BCL modules lying around (which I had bought previously in an attempt to scale up my fibre network from 10G to 25G, but then wasn't able to fing a 25G switch which was silent enough to run it in my flat, so the project was abandoned). These modules are SFP28, and their data sheet says that they support 'up to 25.78 Gb/s bi-directional data links'. Mind here: no specific rates mentioned as in the data sheets of other Finisar modules I have, but rather a range of 'up to', which to me was an indicator that maybe also 10G and even 1G could work.

So … admittedly using multirate with fiberoptic is a bit of a cluster but the good news is that moving forward: the SFP28/QSFP28 modules I’ve encountered are multirate — the spec might require idk

[jabbr]

6 hours ago, dbastin said:

Well, that is a lot better than people who can't theirs to working get ... silence.

@Bertel has given very specific instructions to get it working. Did you follow exactly?

[jabbr]

11 hours ago, Bertel said:

I'd assume that SFP28 modules with their even higher requirements for jitter performance (I think @jabbr you had mentioned that a while back) should even be 'more better' ;-) I don't know about other parameters that might eat up the benefits, like some sort of noise increase due to higher bandwidth or energy consumption, the higher temperature of these SFP28 modules compared to the SFP+ ones (10C more), or what seems to be different Bias and Power levels (see screenshot of SwOS 'SFP' tab in CRS305)...

The improves noise/jitter isn’t the SFP28 module per se, rather the end to end system jitter rejection. Lots of that is implemented on the switch ASIC/FPGA … The system is held to this high standard regardless of PSU or power draw. With fiber, as opposed to copper, the power doesn’t go up with bandwidth to the same extent. If you look at power use per Gb transferred, for example, it’s vastly better. In any case the system noise has to be very very low and the NIC/switch has to make that happen with the power supply that it uses.

 

Also for @Jud, note the data about the SFP+ modules displayed on the screenshot — this data comes from the serial connection

[Bertel]

31 minutes ago, jabbr said:

The improves noise/jitter isn’t the SFP28 module per se, rather the end to end system jitter rejection. Lots of that is implemented on the switch ASIC/FPGA … The system is held to this high standard regardless of PSU or power draw. With fiber, as opposed to copper, the power doesn’t go up with bandwidth to the same extent. If you look at power use per Gb transferred, for example, it’s vastly better. In any case the system noise has to be very very low and the NIC/switch has to make that happen with the power supply that it uses.

… which in turn means that the benefits over pure 10G end-to-end are rather minimal if the hardware (i.e. the NICs in yoC and NAS, and the modules) are designed for up to 25G but everything is running at 10G because the switch (the Mikrotik CRS305 in my case) only does 10G, right?

[jabbr]

1 hour ago, Bertel said:

… which in turn means that the benefits over pure 10G end-to-end are rather minimal if the hardware (i.e. the NICs in yoC and NAS, and the modules) are designed for up to 25G but everything is running at 10G because the switch (the Mikrotik CRS305 in my case) only does 10G, right?

The NICs matter because they pretty much aren’t allowed to emit too much noise. You get the pure end to end 10g jitter rejection but not the even tighter 25g limits … I can’t hear an improvement above 10g. The devices have a circuit which rate reconstructs the incoming data and cleans up the jitter/noise before sending its packet on its way, either out to another device, or to the CPU/cache etc.

[Jud]

13 minutes ago, jabbr said:

I can’t hear an improvement above 10g.

Thanks. Also interesting you *can* hear one with 10g.

[jabbr]

3 hours ago, Jud said:

Thanks. Also interesting you *can* hear one with 10g.

I didn’t say that. I’ve been doing 10g since 2015. I wired my house at that time. I do photography and video editing and those applications notice the speed. They are paying people to take old prograde 10g switches on eBay.  In 2020 during COVID lockdown I upgraded to 100Gbe…
 

I tested several high quality NICs including ultra low latency ones used for high speed trading and can’t tell a difference 10-100Gbe. I have several 1g low powered ARM endpoints … but not a 1g NIC because … why? So I haven’t done real SQ testing of 1g switch against 10g. 
 

The “sound” I expect and achieve from a fiberoptic network is “blackness” on top of which the music plays. There should be no coloration. I’ve achieved that through research about the best quality parts and components. Beyond that, my NAS has 64Gb RAM for fast video editing (cache) and 100Gbe to my workstation etc which serves double duty as my HQPlayer server but aside from being able to run HQPlayer modulators etc my network upgrades haven’t made black blacker. 

[Bertel]

4 hours ago, jabbr said:

The NICs matter because they pretty much aren’t allowed to emit too much noise. 

 

That's great, thank you!

 

Yes, that was my assumption, so I have a Mellanox ConnectX-6 (the CX631102A model) NIC coming which I hope will be a good starting point for settling/solving the SFP28 NIC topic. It has drivers for both Win 11 and Linux available, and I'm hoping HQPlayer Embedded will recognize it (so I don't need to fiddle with Ubuntu).

 

On its compatibility list I see one Finisar module which is the FTLF8536P4BCL. Interestingly its data sheet says it is an SFP+ module. I will get one of these as well, just to be safe, so I have a proven alternative at hand in case my current FTLF1436P3BCL (SFP28 module) should cause issues.

[dbastin]

10 hours ago, jabbr said:

@Bertel has given very specific instructions to get it working. Did you follow exactly?

Thanks for your query.

 

it was @89reksal who gave the excellent instructions, but he showed him self out of this after probably feeling unwelcome due to it being implied he was acting 'entitled'.  I'd say his frustration was misinterpreted ... but we all interpret differently.

 

For example, my comment Well, that is a lot better than people who can't theirs to working get ... silence. was humor, Yoda style.

 

At this point I don't have SFP+ or above modules and trying to gleen which are actually compatible, and of those which sound best, and how to configure RouterOS to suit.

 

SFP28/QSFP28 now seems to solve the compatibility ...  but now wondering if it sounds better than Industrial grade Finisar 1G or 10G.

[Jud]

2 hours ago, jabbr said:

I didn’t say that.... So I haven’t done real SQ testing of 1g switch against 10g. 

 

Thanks for the clarification. I took the wrong implication from your statement that you couldn't hear an improvement above 10G.

 

2 hours ago, jabbr said:

The “sound” I expect and achieve from a fiberoptic network is “blackness” on top of which the music plays. There should be no coloration. I’ve achieved that through research about the best quality parts and components.

 

How have you determined that you have achieved this "blackness"?

[dbastin]

1 hour ago, Jud said:

How have you determined that you have achieved this "blackness"?

Very good question!  A point of reference is necessary but very hard to establish.  I'd say there are a few things that contribute to reducing noise floor and creating noiseless  blackness, and one cant really judge the impact of ethernet noise reduction unless all those other sources of noise have been reduced at least equally and ideally more.   For instance, the difference between one switch and another might not be as distinct if there is high degrees of AC noise polluting the system.

 

The obvious noise sources are AC power, grounding, cabling.  And then there is more abstract sources dealt with by things like Telos Macro G and Q, Synergistic Research ECTs and Tranquillity PODs and Bases, Input caps, Akiko tuning sticks, Bybee, Lessloss Speaker firewall, etc.

[Johnnydev]

1 hour ago, dbastin said:

Zeer goede vraag! Een referentiepunt is noodzakelijk, maar zeer moeilijk vast te stellen. Ik zou zeggen dat er een paar dingen zijn die bijdragen aan het verminderen van de ruisvloer en het creëren van geruisloze zwartheid, en men kan de impact van ethernetruisonderdrukking niet echt beoordelen tenzij al die andere bronnen van ruis minstens even en idealiter meer zijn verminderd. Het verschil tussen de ene schakelaar en de andere is bijvoorbeeld mogelijk niet zo duidelijk als er een hoge mate van AC-ruis is die het systeem vervuilt.

 

De voor de hand liggende geluidsbronnen zijn wisselstroom, aarding, bekabeling. En dan zijn er meer abstracte bronnen die worden behandeld door zaken als Telos Macro G en Q, Synergistic Research ECT's en Tranquility POD's en Bases, Input caps, Akiko tuning sticks, Bybee, Lessloss Speaker firewall, etc.

+1

[Jud]

5 hours ago, dbastin said:

Very good question!  A point of reference is necessary but very hard to establish.  I'd say there are a few things that contribute to reducing noise floor and creating noiseless  blackness, and one cant really judge the impact of ethernet noise reduction unless all those other sources of noise have been reduced at least equally and ideally more.   For instance, the difference between one switch and another might not be as distinct if there is high degrees of AC noise polluting the system.

 

The obvious noise sources are AC power, grounding, cabling.  And then there is more abstract sources dealt with by things like Telos Macro G and Q, Synergistic Research ECTs and Tranquillity PODs and Bases, Input caps, Akiko tuning sticks, Bybee, Lessloss Speaker firewall, etc.

I was thinking of far more prosaic things, such as the mere fact the system is spec'd to work and does work at 10G, where there's a maximum amount of jitter and noise you can have before it won't.
 

I recall in the past @jabbr having made some actual jitter measurements, so I was just wondering if he'd done that at some point with a 10G system, or even better had made before and after measurements when he'd made some change.

[jabbr]

7 hours ago, Jud said:

How have you determined that you have achieved this "blackness"?

When I’ve directly compared copper RJ-45 input to fiberoptic input, the fiber background is very silent/“black”. I also power isolate my entire audio area with a balanced isolation transformer so that the only inputs are a plug to the wall and a fiberoptic cable.

 

I’m not looking for my network to have an SQ, rather no sound.

 

 

[jabbr]

4 minutes ago, Jud said:

I was thinking of far more prosaic things, such as the mere fact the system is spec'd to work and does work at 10G, where there's a maximum amount of jitter and noise you can have before it won't.
 

I recall in the past @jabbr having made some actual jitter measurements, so I was just wondering if he'd done that at some point with a 10G system, or even better had made before and after measurements when he'd made some change.

The difference with 10G is that the spec mandates that the equipment reduce/eliminate incoming jitter via the stressed receiver test. This jitter testing is done by the manufacturers or specialized labs. I certainly can’t jitter test at 25Gbe. Jitter at the DAC is a different issue and while that might affect SQ there is no reasonable path where a 10Gbe jitter isolated network might affect jitter at the DAC

 

… well there is only 1: Ethernet packets. At higher network frequencies, the frequency of network packets is pushed further outside the audio spectrum ie at 100Gbe we’ve got a very brief blip. 

[Jud]

5 hours ago, dbastin said:

And then there is more abstract sources dealt with by things like Telos Macro G and Q, Synergistic Research ECTs and Tranquillity PODs and Bases, Input caps, Akiko tuning sticks, Bybee, Lessloss Speaker firewall, etc.

Some of these noise sources may be so abstract as to be nonexistent. I once caused Synergistic to change its website copy by pointing out in an online comment at Stereophile what complete nonsense it was. What they changed it to was nonsense as well but at least it was more true.

 

(That's probably confusing, so to get into a little more detail: They said they treated cables by hooking them up to a van der Graaf generator and having them undergo quantum tunneling. I pointed out that quantum mechanics, including quantum tunneling, is part of the fundamental fabric of the universe and as such occurs uncountable trillions upon trillions of times in every smallest bit of space, including cables whether or not they're hooked up to spark generators. Synergistic's website was changed within a couple of days to say that hooking the cables to the van der Graaf generator was a "process *we call* quantum tunneling." That's strictly true, they can call the process Harvey if they want, but it doesn't change one iota what's going on - or not - in the cables.)

[Jud]

22 minutes ago, jabbr said:

there is no reasonable path where a 10Gbe jitter isolated network might affect jitter at the DAC

Once the signal gets past the Ethernet connection it's out of the purview of specs and tests, so inside the computer that converts the signal from Ethernet to USB (at least I assume that's the typical path for most here) I suppose there could be something in a higher speed signal vs. a lower speed one that might affect noise from that component into the DAC, or might affect it at the receiving end internal to the DAC.

 

For example, I recall the reason it was said the ER had a 100M interface was because at the time there weren't the associated 1G components with the electrical noise characteristics (and presumably price) needed.

[jabbr]

2 hours ago, Jud said:

Once the signal gets past the Ethernet connection it's out of the purview of specs and tests, so inside the computer that converts the signal from Ethernet to USB (at least I assume that's the typical path for most here) I suppose there could be something in a higher speed signal vs. a lower speed one that might affect noise from that component into the DAC, or might affect it at the receiving end internal to the DAC.

 

For example, I recall the reason it was said the ER had a 100M interface was because at the time there weren't the associated 1G components with the electrical noise characteristics (and presumably price) needed.

 

The end-to-end testing mandated by the 10G specifications includes testing at the receiver so at the computer if the receiver is SoC, or at the NIC if the receiver is a NIC. I've not seen a shred of actual measurements comparing noise at the board level between 100m and 1g or 100g for that matter.

 

At 100Gbe you could easily transmit the entire song in the short gap between tracks so the network wouldn't be an issue!

 

That said it is reasonable to use a low powered, low noise endpoint like the Fitlet or the Clearfog or the OpticalRendu ... I think its very reasonable for something like the OpticalRendu to be designed for low noise and to optimize the USB output. My own impression is that there may be factors on the board that affect SQ via the USB output and use LPS etc for my own endpoints. 

 

The discussions I've had regarding blocking upstream factors via 10Gbe+ e.g. servers, NAS have to do with the mandated stressed receiver testing, and see no reason why downconverting the signal to the endpoint should be an issue -- this is what I've been doing -- my low powered ARM SBC 1g endpoints sound just as good as the 10g NAA that I had built using a Celeron board. The sensitivity of DACs to USB output also is DAC dependent to my ears.

 

Does the OpticalRendu do 100m Ethernet via fiberoptic?

[dbastin]

2 hours ago, Jud said:

Some of these noise sources may be so abstract as to be nonexistent.

I use all those I listed except the Lessloss firewall (which you can read reviews about). In my experience each does 'something' which contribute's to improving sound quality.  For example my CRS 305 has an ECT on the switch chip, is grounded via HD ground cable to a Synergistic Research Active Ground Block SE, and sits on a Tranquility POD.  It didn't take me long to recognise the benefit of each of these - despite the 10G specs.

 

3 hours ago, jabbr said:

I’m not looking for my network to have an SQ, rather no sound.

Agreed, however I am not certain what zero coloration or impact from ethernet will sound like, if it is actually possible!

 

A Devialet Pro is my endpoint, connected via wifi to a battery powered low power WAP (so completely floating from the network).  But still, changes upstream are detectable, even upstream of the CRS 305.

 

I even hear improvements from most roon updates, such as the one installed a few hrs ago.

[jabbr]

1 hour ago, dbastin said:

I use all those I listed except the Lessloss firewall (which you can read reviews about). In my experience each does 'something' which contribute's to improving sound quality.  For example my CRS 305 has an ECT on the switch chip, is grounded via HD ground cable to a Synergistic Research Active Ground Block SE, and sits on a Tranquility POD.  It didn't take me long to recognise the benefit of each of these - despite the 10G specs.

 

Agreed, however I am not certain what zero coloration or impact from ethernet will sound like, if it is actually possible!

 

A Devialet Pro is my endpoint, connected via wifi to a battery powered low power WAP (so completely floating from the network).  But still, changes upstream are detectable, even upstream of the CRS 305.

 

I even hear improvements from most roon updates, such as the one installed a few hrs ago.

Well certainly software such as Roon which does filtering and modulation certainly would affect SQ entirely aside from the network hardware. 

 

My focus on SQ has had to do with optimizing filters and modulators in HQPlayer and my NAS and Roon server are connected to my HQPlayer server via 100Gbe … doing DSD1024 with advanced modulators is now possible with the latest set of processors! … the Roon server handles streaming via Qobuz etc. 
 

Likewise digital crossovers, room correction and arrays of speakers can have a vast effect on SQ! One could even do things like power DAC etc etc.

 

I remain confident that with the solid Ethernet backbone these applications can be best tuned for SQ.