Optical Network Configurations

[jabbr]

39 minutes ago, Crom said:

I was reading earlier comments about SFP28 being downwards-compatible with 1G/10G etc. I'm looking at various motherboard options for a new music server and this one looks interesting because it has 2 x 25G optic cages built in: https://servers.asus.com/products/servers/server-motherboards/K14PA-U12#Features

 

I'm interested in Cisco AOC cables or similar and wondered whether I can just use a standard 25G to 25G like this: SFP-25G-AOC5M Cisco 5M SFP28 to SFP28 Active Optical Cable (https://serversfit.co.uk/sfp-25g-aoc5m-cisco-5m-sfp28-to-sfp28-active-optical-cable.html?gad_source=1&gclid=EAIaIQobChMItMPj6YHqhQMVpZNQBh2Axwq7EAYYBCABEgJaUfD_BwE) or do I need to find a cable that takes care of the "downspeed" eg 25G to 1G?

Thanks for any help.

It depends on where you want the network connection to go. If you are going 25->25 then an AOC should work (assuming its compatible with your SFP28 cage). I have no idea of a 1G SFP device will handle a 25G AOC cable. Its not that the cable cant do 1G its that the 1G SFP cage may not work with the 25G AOC cable

 

You are typically safer connecting a 25G SFP28 via e.g single mode to a 1G SFP

 

Now this also depends on the driver built into your motherboard which *should* autosense but might need to be manually set to 1G

[jabbr]

1 hour ago, Crom said:

It would have been helpful if I'd provided what will be hanging off the other end. In the back of my mind was the fact that I would be plugging this into an etheregen!

The question is, then, whether the EtherRegen will accept/work with a SFP28 AOC cable... it might or it might not. 

[jabbr]

52 minutes ago, SQFIRST said:

 

Very unlikely that 1G cage will accept a 25G module. The form and dimensions of quad speed modules appear to be slightly bigger and different than sfp/+.

 

 

Its not that the QSFP28 modules are quad speed, its that they are essentially 4x SFP28 co-packaged. To operate over a single mode LC-LC duplex cable the QSFP28 100Gbe module has to multiplex 4 channels, for example by using 4 different laser frequencies each carrying 25G

[jabbr]

15 minutes ago, audiom3 said:

What would be gained in going to 25G?  The only thing that I can think of is maybe, lower jitter?  But I cannot find anything conclusive regarding jitter spec.  Latency and/or bandwidth aren't an issue with 10G for home use.  

 

Of course the required jitter specs are published as part of the stressed eye and its conclusive that 25G requires a tighter spec than 10G. Unsure that matters for us though. The 25G spec is the first optical spec that includes multirate support ie 25,10,5,2.5,1 G in the same device --- multirate has been present in copper ethernet forever.

 

None of this is actually new, Ive been using 100Gbe i.e. QSFP28 at home for some years now, its just that the costs are getting down to reasonable for most people. I've also posted for a long time that I can't hear a difference in SQ going to 25 nor 100Gbe

 

This *is* a good time to make the point that the 100Gbe devices which require vanishingly low jitter in order to work, DO NOT use special power supplies nor clock crystals, rather the clocking electronics have become rather sophisticated

[jabbr]

21 hours ago, audiom3 said:

Thanks @jabbr.  Very helpful info.  I will say, I thought I was near the pinnacle of SQ through fiber optics until I added an LHY SW-10 switch.  It (re?)clocks the optical ports too and not just the copper ports.  And what a huge difference it made to SQ to clock my fiber lead right before going into my Lumin U2 sfp cage.  Best tweak I've bought to date.

 

You know completely honestly in this hobby when you hear something more pleasing to your ear you don't need to question it.

 

My posts are directed entirely at the theory of why something might sound better. I'm just using what we know about e.g. 100Gbe and how it works and what optoelectronics make it work. 10Gbe and newer require jitter measurements via stressed eye pattern testing and 1Gbe and older don't. My 100Gbe PCI-e card outputs vanishingly low jitter from my server despite bog standard PSU so really Mellanox nee NVidia in my case has performed a feat of engineering. Thats all really. 

 

Also to be clear, my audio area is protected by a heavyweight balanced power wall and cheap switching PSUs in my house, my fridge etc can't reasonably/appreciably penetrate that iron. If I had an ethernet switch within my audio power domain, then yeah all sorts of stuff becomes important.

[jabbr]

21 hours ago, SQFIRST said:

This is good to know. What changes did you experience going to 100GBE devices with regards to power domain separation/control? I imagine your audio is still protected behind isolated power and clean power at the endpoint?

Right, only optical and isolated power into my audio domain. No 100Gbe devices, only the audio endpoints -- I *have* used 10Gbe PCIe cards when I was building PC based endpoints. The endpoints are within the audio power domain and output USB etc so its reasonable to optimize them. That said @Jud recently posted the fiberoptic USB cable experience!

[jabbr]

20 hours ago, audiom3 said:

Yes, until someone like @jabbr posts about 100G tech and the lack of need for re-clocking 🤣 I am wondering how much better 100G is vs 10G and if upgrading a switch could do exactly what the OCXO in the LHY is doing.  Guess I will have to try a new switch at some point to see.

 

The fast switches already use stuff like this https://www.renesas.com/us/en/products/clocks-timing/femtoclock-low-phase-noise-frequency-synthesizers

[jabbr]

2 hours ago, TRHH said:

@jabbr Thanks for sharing your expertise here. 

 

As my current switch - Cisco Meraki Go GS110-8-HW-EU - and SFP - Finisar FTLF1318P3BTL - are End-Off-Life, the time has come for an replacement (upgrade).

 

A new switch with dual rate (SFP/SFP+), 8-12 ports, no need for PoE, fanless, and some new SFP/SFP+?

 

THANKS for any suggestions and have a nice WE

 

Torben

Honestly there is no one right answer.

 

I have used the 4/5 port Mikrotik 10g/1g switch with Mikrotik dual 10g/1g SFP+ modules. It is fanless and inexpensive. I got it when it was the new kid on the block and haven't compared other fanless 10g switches etc. I recommend it because it works for me.

 

I'm unsure if there is a 25G switch similar to this.

 

Oh look https://www.servethehome.com/mikrotik-crs510-8xs-2xq-in-review-8-port-25gbe-2-port-100gbe-switch-marvell/ (fanned)

 

[jabbr]

3 hours ago, TRHH said:

Thanks for you comments.

 

The red ones are the ones that need to be replaced/upgraded:

 

 

Have a nice WE

 

Torben

 

Frankly with the best isolation, the device closest to your DAC/endpoint is the only one that really matters: loose the LevelOne-GVT FMC and replace with something from (for example) Uptone or Sonore  -- although 10Gbe devices are known (required) to have low jitter, theres also the issue of common mode noise transmission over copper ethernet which is really why we started with fiberoptic in the first place.

 

I have used RJ45-SFP+ modules which work well for networking but I HAVE NOT used these with audio endpoints so can't say how something like the Mikrotik would sound using one of these to your device -- you could test this and tell us!

[jabbr]

1 hour ago, audiobomber said:

I agree, dual SFP ports are a great feature, if you need or can benefit from them. As are the generous number of ports in the SW-10.

 

There is reclocking in the ER, and in every switch as far as I understand, and also in any decent FMC. Maybe all FMCs? Anyone?

 

1) every switch I know does "reclocking" because SERDES

2) the topic of clocks for network switches is more complicated that simply the "OXCO" lingo that is thrown around

3) it has been conclusively demonstrated that the overall "jitter" of the switch, measured in femtoseconds for example, can be drastically better than the jitter of the quartz crystal used to clock the clock generator silicon itself

4) the incoming ethernet signal must be phase locked against the synthesized clock which controls the switch chip itself. in a multiport switch each incoming signal will have a different phase and the chip needs to account for this i.e. lock onto an input signal coming from an asynchronous clock domain

5) ummm so right this OXCO is largely a bunch of marketing (for ethernet switches)

6) buy what you want and listen to what you like

[jabbr]

7 hours ago, SQFIRST said:

However, there are also some carefully reported findings with well described instances where it does merit some room to for the possibility that at least some more definition is needed.
I agree that the focus needs be less about the ocxo and more about what is causing the better sounding ethernet signal in these instances.

Ok … however the purest measured Ethernet signals, having the lowest jitter, comes from equipment which does not use external clocks. So whatever the magic wand does, has never been demonstrated to improve a modern Ethernet signal. 1Gbe is frankly ancient at this point and 10Gbe is a previous generation. Modern Ethernet electronics has come a long way and an external (OXCO) clock has not been shown to improve the Ethernet signal. 

[jabbr]

5 hours ago, Superdad said:

 

Those Renesas clock synths are nice, though a bit pricy at $16pc. in quantity, and we don't really need 3 inputs and 8 outputs.  The Silicon Labs (now Skyworks) synth we use in the EtherREGEN has phase jitter specs equal to the Renesas, but the Analog Devices synths we are moving to for EtherREGEN Gen2 have dramatically lower phase noise that any of the others.

So one does not need to go to 10G/100G switches to get that level of clocking performance. 

 

Of course this leaves out the noise/jitter that gets embedded by optical transceiver modules (differences in which I believe is the primary cause for the variations folks here between SFP brands/models; and why Finisars generally end up as preferred).

 

 

 

👏 👏 👏 Yes yes, my point is that these devices are actually more important than the oft quoted OXCO regarding ultimate jitter. Analog Devices is a great company of course and the reported jitter on some of these devices is crazy low.

 

Its not that you have to go to 10G/100G rather that these specs require compliance testing assuring that the devices behave. There is nothing preventing a specific 1G switch/device from being implemented to achieve the behavior ie stressed receiver eye pattern of the 10G spec. If you have a specific 1G device which you love the "SQ" of then you are all set. Nor is there a guarantee that any 10G switch will sound better to you. 

 

and as you further note, the clock is only part of the entire jitter budget. the 10G/100G and all the newer specs don't need to focus on requirements for each individual component because the specifications are end-to-end. specifically the specs require that the receivers accept a marginal input and clean it up.

 

in the 10G/100G (i.e. newer) world the end user need not care about which clocking device, crystals etc are used nor can they change them. external clock inputs don't exist in the lowest jitter switches (and of course these switches aren't intended for the home user)

[jabbr]

2 hours ago, SQFIRST said:

That seems like a sensible approach doesn't it? It may pose some engineering complications with regards to sensitive audio design. If that is not the case then it is a no brainer. Just from an adoption standpoint, there will be resistance as the norm has been 1G and lower up until now. Add pricing increase to that, the case becomes quite difficult. But certainly a ground breaking design waiting to happen.

 

Its entirely unclear to me that the endpoints need to handle 10G ethernet given the assumption that the switch device which is capable of 10G would presumably perform in a similar fashion when operating at 1G

 

2 hours ago, SQFIRST said:

 

Which brings us back to the earlier discussion about the advantage of using 10G and higher protocols to remove jitter.

 

Its not the 10G protocol which removes jitter, rather the specification which requires end-to-end testing with a stressed receiver i.e. a crappy input and clean output. The implementations are required to remove jitter

 

2 hours ago, SQFIRST said:

The point being that a concerted effort targeting digital 'noise' removal, which could include multiple specialized solutions, maybe needed for quality output. One approach does not have to negate the other. Also, this type of effort may not be for everyone as people have differing levels of patience, idea of fun and interests. 

 

Again all compliant 10G+ switches remove jitter and all fiberoptic switches remove common mode noise. You don't need to worry about this. 

 

2 hours ago, SQFIRST said:

 

I have the X1 and did notice the same issue and narrowed it to the transceiver locking mechanism. The Lumin did not allow for the lock to open properly which prevents the module from seating into the cage. I did get a generic 10G to work with it but did not like the performance. It is a good idea to not try to make it work and heed Lumin's recommendation.

 

Its not the 10G SFP which removes jitter though any device attached to the DAC might affect SQ.

[jabbr]

5 hours ago, audiom3 said:

I couldn't get the 10G SFP+ to work in the Mikrotik CRS309

Is the 10G module you tried on the Mikrotik CRS309 compatibility list? Although some lore has developed about certain  SFP modules, they aren't required to work when not on the compliance list. One of the things about having boxes of SFP modules is that you can usually find one that works.

 

[jabbr]

9 hours ago, SQFIRST said:

Based on your statements it would seem that the Mikrotik CRS305 is not compliant.

You are confused about what I’m saying. 
10G+ Ethernet compliance testing is specified (in the specification itself) Compliant means it passed the required testing. 

[jabbr]

9 hours ago, audiom3 said:

It wasn't that the transceiver wouldn't work in the 309. It was that it would not work with the CRS309/Lumin combo.  I use Finisar 1471s all over my house. Although a lot less now that I've ridded my AV components of FMCs.

You don’t need to use the same SFP(+) module in the switch and endpoint 

[jabbr]

9 hours ago, dbastin said:

For what its worth, I recently compared CRS 305 to ER.

If you are dealing with copper outputs/networks … there is no reason to think that inexpensive … or any … 10G switches have low common mode noise and SQ can be affected by that … essentially fancy ground loops. 
 

I have done exactly zero testing of 10G switches output using copper to endpoints 

[jabbr]

31 minutes ago, audiom3 said:

I am aware of that now. I had to experiment to figure it out. That's why I'm using a 1471 in my switch and a 1318 in my Lumin. 

 

Thats great. The actual reason that I started using the 1471 was that it was on the Intel NIC compatibility list and one of 2 OEMs for Intel. The first Intel branded SFP+ module I purchased was Finisar. At the time I had a Brocade VDX 10G switch and it only accepted Brocade branded SFP+ modules. Brocade was aquired by Broadcom which was then acquired/merged with Avago the other major SFP+ OEM.

 

In the old days and still with lots of equipment you had to use a branded SFP+ module which each brand of equipment and the modules have a code. Nowadays fs.com can supply coded OEM modules but if you try to use a ... for example ... Cisco coded SFP module in a Brocade switch it just won't work even if the SFP modules are electrically the same. Hence look at the compatibility list of whatever equipment you are using.

 

I have no idea where Mellanox gets its (Q)SFP modules from but I try to use those with my mellanox switches and NICs.

 

For endpoints I do try to use Finisar

[jabbr]

11 minutes ago, audiobomber said:

As long as the wavelength is the same?

 

Yes yes, SR with SR and LR with LR and any of the variations of these need to be matched at each end. I avoid the ER and ZR modules

[jabbr]

20 minutes ago, SQFIRST said:

 

@jabbr I really appreciate your efforts to right track information. I am not questioning ethernet standards and have no confusion with that. Just need some clarity on the theory in practice.

 

My actual usage experience with the Mikrotik CRS305, which is central in my network, is not following the assumption that you have mentioned. There is a clear difference when the switch is connecting at 10G end to end vs. when connecting at 1G. This difference is only based on my hearing the output from audio setup and not by any other measurement. To be clear, the CRS305 does not appear to be performing in the similar fashion at 1G as with 10G. In my experience so far, unless the connection is running with both of the ends operating at 10G, the benefit is not present in the audio in my chain. 
 

 

Ok I understand and indeed since the compliance testing is done at 10G I am indeed assuming that the output at 1G would have similar performance ...  but not necessarily 

 

Your system may be more complicated than mine in some sense. My fiber goes directly to my fiberoptic endpoint at 1G and from the endpoint to the DAC over USB. I have made 10G endpoints i.e. a PC but really like these low powered ARM devices as NAA.

 

If you are using a fiberoptic Lumin at 1G how does the switch connect at 10G? 

 

Really its the last connection to endpoint which I feel is most important and if this comes from a 10G switch then I'm assured that any upstream jitter doesn't pass ... and has been tested not to pass

 

I mean if these SFP modules and power supplies and etc etc are so important, why not set up the network correctly in the first place?

[jabbr]

22 minutes ago, SQFIRST said:

This is exactly what it comes down to and it would be great if the CRS305 handled 1G as it does 10G but that is not the case. The option to connect 1G optical from the switch to the Lumin is good but in my case, I get better performance using the EtherRegen as the final 1G device.

 

My endpoint setup is simple as there is only network input terminating at the DAC (X1). It is a different experience from using USB to DAC.

The Solid-Run CN9132 Base is the only 10G fanless / low powered endpoint that I am aware of. Of course this endpoint would use USB output.

 

I might test this with a fiberoptic USB cable but frankly Im not sure I could tell the difference between this and my A388 device... I might need to upgrade my DAC to something that can handle DSD1024/DSD2048 or  more channels etc

[jabbr]

24 minutes ago, SQFIRST said:

I suspect your mellanox or the 100g devices you have in place offers a high level of compliance where you do not see the similar issues that us mere mortals have been reporting. If you really want to test things you may end up working backwards a bit :)

 

No, I just try to keep things simple because my brain gets totally fried trying to listen to too many combinations of widgets, cables, power supplies etc. I've been through upgrade hell many times. 

 

I actually realized that I've had a CN9132 Base for a year that I haven't had the time to look at.  🥺 When you have to connect and disconnect devices from the network and redirect where everything goes I find it hard to A/B test.  On top of that the HQPlayer settings demonstrate real SQ changes... I think I'd honestly need to upgrade my DAC in order to hear very subtle differences in anything else at this point e.g. something like the Holo May

 

My goal is to make the network entirely invisible

[jabbr]

9 minutes ago, Jud said:

 

When (if) ever Comcast gets around to bringing fiber to the home here, I checked and they're multimode. So I decided when building the house to have everything done multimode to avoid any possible incompatibility. That's what the entire network is now. So I'm just accustomed to looking at multimode.

 

Any problem at all with making the final connection from switch to UPnP/NAA endpoint single mode?

You are highly unlikely to ever want more than more than 25G to 100m so multimode is fine. There is zero problem either taking multimode in from the street or from your house on a switch and then combining with singlemode out ... you can also go to your NAA endpoint in multimode without loss of signal quality. I standardized on singlemode in my walls because singlemode scales up to 100g and beyond. Multimode cables for 100G are ridiculously complex and expensive.

 

If you can hear the difference between a single and multimode fiber cable you have the ears of a nubile fairy!

[jabbr]

19 minutes ago, taipan254 said:

 

 

Sorry for jumping in here, but I am seeking a point of clarification as I think about my own network evolution. From earlier in this thread, I was under the impression that at least one of the connections had to be at 10G for the benefits of 10G to be realized. For example, I was under the impression the below scenarios would perform identically (let's call this "Impression 1"):

 

Internet ->1G Copper-> MicroTik Router ->10G Copper -> Music Server ->1G Optical-> Moated End Point -> USB-> Moated DAC

Internet ->1G Copper-> MicroTik Router ->10G Optical -> Music Server ->10G Optical-> Moated End Point -> USB-> Moated DAC

Internet ->1G Copper-> MicroTik Router ->1G Copper -> Music Server ->10G Optical-> Moated End Point -> USB-> Moated DAC

 

All provide a moat / isolation for the end point. All incorporate 10G (copper or optical; 1G or 10G optical always at endpoint).

 

Now I think I'm interpreting the conversation as "As long as you have a 10G router, even if you have everything on it running at 1G, you will still get the benefits of the 10G specification" (let's call this "Impression 2"). 

 

Is Impression 1 right or wrong? Is Impression 2 right or wrong? 

 

I'm just here to iron out any confusion I'm having. The router and endpoint upgrades are next on my to-do list.

 

 

 

I don't think either assumption is necessarily true.

If the Music Server is connected by fiberoptic with the switch at 10G then upstream noise (from the internet and server ) is blocked.

If the switch is connected to the endpoint by fiber then common mode noise from the switch is blocked.

[jabbr]

1 hour ago, dbastin said:

I think this assumes all noise (and possibly the impacts of it) can be blocked by fibre or the 10G spec.  I'm not sure that is the case.

There are actual measurements. Upstream noise isn’t transmitted. Whatever SQ differences you hear are not via the Ethernet stream transmitted along 10G+ fiber and most certainly not at 100G.

 

what you hear is via a different mechanism and there are several possibilities. some of the possibilities are electrical.