Optical Network Configurations

[jabbr]

On 1/14/2023 at 6:45 PM, SQFIRST said:

I find that the Roonserver is similar to any source component in the system and benefits from good power supply and careful setup. I am open to learning new things and changing course that creates better listening experience so I am interested in hearing any different experience others have.

 

 

My server that Roon runs on is a Dell Poweredge in which I have installed a Mellanox 100Gbe NIC. This NIC is powered by a bog standard PCIe bus/PSU. The low noise/jitter requirements to operate at 100GbE are a marvel of modern technology. These NICs have been engineered with their own onboard power systems and are not dependent on a fancy PSU. Similarly modern PCIe buses at PCIe-4/5/6 require vanishingly low noise/jitter in order to work, and they work. Admittedly common mode noise can be generated but again the fiber is a perfect solution for that.

[The Computer Audiophile]

45 minutes ago, jabbr said:

 

My server that Roon runs on is a Dell Poweredge in which I have installed a Mellanox 100Gbe NIC. This NIC is powered by a bog standard PCIe bus/PSU. The low noise/jitter requirements to operate at 100GbE are a marvel of modern technology. These NICs have been engineered with their own onboard power systems and are not dependent on a fancy PSU. Similarly modern PCIe buses at PCIe-4/5/6 require vanishingly low noise/jitter in order to work, and they work. Admittedly common mode noise can be generated but again the fiber is a perfect solution for that.

Makes me wanna try 100Gbe

[SQFIRST]

@jabbr thank you for sharing. It is an impressive setup. I can certainly appreciate the jitter control and subsequently, better audio quality aspect in your implementation. My current network state is a simple home user setup. Now that I know about the tech in the Mellanox it will certainly be on my radar. Thank you again for your pointer on balanced power isolation as I implemented that on the power section for my roonserver and it is significantly quieter now.

[juniorbudel]

6 hours ago, dbastin said:

If you change a cable or power supply on your dirty side, can you hear any difference?

I still haven't made an AB, today I have a Wireworld Chroma 7 between the server and the router.  I will do this test.  

 

The problem is that all the other network cables on the dirty side are common, they will certainly narrow the differences, if any.

[jabbr]

2 hours ago, The Computer Audiophile said:

Makes me wanna try 100Gbe

I did it because ... I could ... and already had a run of single-mode fiber from my basement to my office. Video/photo editing might see a difference. I tested various 10,25,40 and my 100Gbe NICs and honestly can't hear a difference. The 100Gbe does have infinitesimally low jitter/noise and that fact that it works, and the fact that it doesn't require any special PSU for the PCIe bus powering tells me that Mellanox (and other NIC vendors) must have engineered impressive on card power. No doubt ultra low noise linear power chips behind ultralow noise voltage converters etc. My 100Gbe switch plugs into the wall, and similarly although you can do 48V DC power, it all gets voltage converted anyways. There's no ability to externally clock this stuff and since the phase noise requirements are vanishingly low really there is no need nor way to optimize it further.

 

But by all means I'd love for someone with great hearing to replicate my results!

[The Computer Audiophile]

18 minutes ago, jabbr said:

I did it because ... I could ... and already had a run of single-mode fiber from my basement to my office. Video/photo editing might see a difference. I tested various 10,25,40 and my 100Gbe NICs and honestly can't hear a difference. The 100Gbe does have infinitesimally low jitter/noise and that fact that it works, and the fact that it doesn't require any special PSU for the PCIe bus powering tells me that Mellanox (and other NIC vendors) must have engineered impressive on card power. No doubt ultra low noise linear power chips behind ultralow noise voltage converters etc. My 100Gbe switch plugs into the wall, and similarly although you can do 48V DC power, it all gets voltage converted anyways. There's no ability to externally clock this stuff and since the phase noise requirements are vanishingly low really there is no need nor way to optimize it further.

I’m just playing 3-8GB files and want them to start as soon as I press play. Faster speed = good :~)

[jabbr]

 

2 hours ago, SQFIRST said:

@jabbr thank you for sharing. It is an impressive setup. I can certainly appreciate the jitter control and subsequently, better audio quality aspect in your implementation. My current network state is a simple home user setup. Now that I know about the tech in the Mellanox it will certainly be on my radar. Thank you again for your pointer on balanced power isolation as I implemented that on the power section for my roonserver and it is significantly quieter now.

 

Sure, the other NIC's we have talked about (besides Intel) are Solarflare which were designed for ultralow jitter. Xilinx acquired Solarflare and AMD acquired Xilinx  (NVidia acquired Mellanox). Both have new "Smart" NICs which incorporate ARM cores (and perhaps FPGA) on the NIC itself to further offload processing from the CPU. Great stuff for NVMEoF and AI/inferencing etc...

 

Again, they all sound great to me and I can't detect a difference in SQ.

 

I'd remove the Roon server from your audio area and use the balanced power for your audio area... my 2p I've also put my equipment on ball bearing / cup based vibration isolators so 😝

[jabbr]

15 minutes ago, The Computer Audiophile said:

I’m just playing 3-8GB files and want them to start as soon as I press play. Faster speed = good :~)

Yes yes and the mainstream application for 100Gbe is NVMEoF. 25Gbe on an SFP28 module is easy. 100Gbe uses Quad SFP28 (QSFP28).

 

I'm (obviously) not at all opposed to speed and wanting my multi-gig photos/videos to open/edit faster was a motivation, but I don't want folks to think that upgrading will improve SQ ... there really is a limit to how many femtoseconds ... or god forbid attoseconds ... that one can hear :sarcasm:😏

[jabbr]

My current setup

 Mellanox ConnectX-5 on my office workstation because it has enough power to supply my QSFP28 100GBASE-LR4 modules.

 Mellanox ConnectX-4 on my server (which is right next to the switch and is connected via QSFP28 fiber cable.

 Mellanox ConnectX-4 on NAS

[The Computer Audiophile]

1 minute ago, jabbr said:

My current setup

 Mellanox ConnectX-5 on my office workstation because it has enough power to supply my QSFP28 100GBASE-LR4 modules.

 Mellanox ConnectX-4 on my server (which is right next to the switch and is connected via QSFP28 fiber cable.

 Mellanox ConnectX-4 on NAS

Which NAS do you use? Your own build?

[SQFIRST]

53 minutes ago, jabbr said:

I'd remove the Roon server from your audio area and use the balanced power for your audio area... my 2p I've also put my equipment on ball bearing / cup based vibration isolators so 😝

 

It is upstream of my audio chain. I understood what you are saying and appreciate your help and great information, thanks again. Ball bearing isolation is really good.

[jabbr]

37 minutes ago, The Computer Audiophile said:

Which NAS do you use? Your own build?

Its a low power Xeon L3426 (ancient) on a supermicro board, with an LSI SAS3 card (LSI 9300-8e). The external SAS3 expander enclosure(s) are pc-pitstop 15 bay (https://www.pc-pitstop.com/15bay-12g-sas-expander-enclosure)

 

runs Ubuntu server 22.04 which include ZFS

[Jud]

7 hours ago, The Computer Audiophile said:

Makes me wanna try 100Gbe

 

So let it be said, so shall it be done.  😉

[dbastin]

9 hours ago, The Computer Audiophile said:

Makes me wanna try 100Gbe

Me too!

I mentioned previously I am using Mikrotik CRS 305 (10GBe) as my router and it is easily better than the Ubiquiti EdgeRouter X SFP (1GBe) it replaced.  But ... when I installed a EtherRegen as FMC instead of a generic FMC immediately upstream of the CRS 305 that provided a further improvement.  So, it suggests perhaps even the CRS's 10GBe low jitter does not create immunity to upstream changes.

 

I am also curious if a low jitter NIC can be so beneficial that the performance of the server (in terms of noise etc) is not relevant.

 

My setup in simple terms is ...

 

Broadband service > EtherRegen > fibre > CRS 305 > Server (Antipodes) > WAP > Devialet

 

Maybe there should be a new thread about the benefits of 5Gbe, 10GBe, 100GBe and how best to gain the most benefits from these.

[dbastin]

8 hours ago, juniorbudel said:

I still haven't made an AB, today I have a Wireworld Chroma 7 between the server and the router.  I will do this test.  

 

The problem is that all the other network cables on the dirty side are common, they will certainly narrow the differences, if any.

How about compare UTP cables with STP cables.

[jabbr]

20 hours ago, Jud said:

 

So let it be said, so shall it be done.  😉

https://www.servethehome.com/mikrotik-crs504-4xq-in-review-momentus-4x-100gbe-and-25gbe-desktop-switch-marvell/3/

 

gulp

[The Computer Audiophile]

12 minutes ago, jabbr said:

https://www.servethehome.com/mikrotik-crs504-4xq-in-review-momentus-4x-100gbe-and-25gbe-desktop-switch-marvell/3/

 

gulp

Hey @jabbr, do you know of a card that supports 100Gbe on macOS?

[jabbr]

25 minutes ago, The Computer Audiophile said:

Hey @jabbr, do you know of a card that supports 100Gbe on macOS?

Welcome to the vbox/qemu twilight zone hell...

[The Computer Audiophile]

1 minute ago, jabbr said:

Welcome to the vbox/qemu twilight zone hell...

 

https://www.atto.com/products/ethernet-adapters

 

[The Computer Audiophile]

ouch.

 

[The Computer Audiophile]

Can do Thunderbolt to 25Gbe, but now that 100Gbe is more normal, who wants 25 🤣

 

https://store.atto.com/thunderbolt-adapters/213-dual-40gb-thunderbolt-to-dual-1025gbe-adapter-sfp28s-included.html

 

[jabbr]

13 minutes ago, The Computer Audiophile said:

ouch.

 

 

Woah!! Honestly I find the Mellanox prices vastly better. I think at that price its really worth it to get say a Dell workstation to fool around with. Notice the number of PCIe lanes and you need a motherboard with one for the GPU and another for the NIC. 

 

You can do an ebay search for ConnectX-4 100Gbe (or ConnectX-5) and look for refurb if you don't want used. Unless you have a specific application, I've used ebay to fool around with lots of stuff and have had great results. For example, I paid roughly 1/20th the retail for my Mellanox SN2700 switch

[m.i.c.k.e.y]

Help

 

Anybody had a comparison on SQ between these transceivers:

- Finisar FTLF1323P1BTL Transceiver SFP Fast Ethernet SM 100Mbit/s 100BaseFX LC-Duplex

- Finisar FTLF1318P3BTL Transceiver SFP Gigabit SM 1000Mbit/s 1000BaseLX LC-Duplex 

both are Singlemode OS2

 

Thanks!

[jabbr]

https://mikrotik.com/product/crs504_4xq_in

 

Anyone who wants to move to 100Gbe ethernet, this switch is very cost effective and fairly easy to set up. Its a bot finicky with AIO fiber cables (but I didn't spend a great deal of time trying to get working). My 100Bgase-LR4 QSFP28 (single mode) work great, also x4 SFP28 breakout cables.

 

 

[The Computer Audiophile]

1 minute ago, jabbr said:

https://mikrotik.com/product/crs504_4xq_in

 

Anyone who wants to move to 100Gbe ethernet, this switch is very cost effective and fairly easy to set up. Its a bot finicky with AIO fiber cables (but I didn't spend a great deal of time trying to get working). My 100Bgase-LR4 QSFP28 (single mode) work great, also x4 SFP28 breakout cables.

 

 

I was looking at this really hard last night!