Optical Network Configurations

[pl_svn]

15 minutes ago, jabbr said:

use the dual 10g/1g devices on the switch side

 

so... that Mikrotik S+AO0005 cable will (ok, ok... should! 😉) work between either a 1Gb SFP (TP-Link) switch and EtherRegen or a Mikrotik SFP+ switch (CSS610-8G-2S+IN or CRS305-1G-4S+IN) and the EtherRegen, right?

 

I'm dumb, I know 😛

[jabbr]

1 minute ago, pl_svn said:

 

so... that Mikrotik S+AO0005 cable will (ok, ok... should! 😉) work between either a 1Gb SFP (TP-Link) switch and EtherRegen or Mikrotik SFP+ switch and the EtherRegen, right?

 

I'm dumb, I know 😛

It advertises itself as dual so likely would. Really the interactions between all these devices and the endless combinations of devices at both ends is so large that a list of known good configurations is essential. Let us know!

[pl_svn]

... ok: ordered a Mikrotik CSS610-8G-2S+IN switch and S+AO0005 cable

unfortunately delivery, here, will take a couple of weeks 🙄

will report how it goes with the EtherRegen 😬

[R1200CL]

@jabbr

We wasn’t, at least I wasn’t, adding 1\10 options into the possibly way of mixing switches and SFP’s

There discussion was using specific two Finisar modules. One pure 1GB (FTLF1421P1BCL) and one pure 10GB (FTLX1475D3BTL)

I did one time suggest to use the 1\10 possibility, but @Superdad clearly stated that wouldn’t help as no auto negotiation would occur. So there was no reason for further discussion in that matter. (Until you added this option again). 

 

We are (at least I am) discussing the 2 Mikrotik switches where either a mix of those two specific modules will work or not. And in addition Mikrotik own cable, which to my understanding is a pure 10GB SFP+ connection.

 

There are 4 possible ways to mix this.

• FTLX1475D3BTL and FTLX1475D3BTL 10>10 (This test also cover Mikrotik cable)
• FTLX1475D3BTL and FTLF1421P1BCL. 10>1
• FTLF1421P1BCL and FTLF1421P1BCL.  1>1
• FTLF1421P1BCL and FTLX1475D3BTL  1>10

 

The other possible options has been explained in AfterDark’s tread. And understood by most I think. For everyone that doesn’t have a 10GB switch the FTLX1475D3BTL will work. Simple as that. (Until that guy with a very specific device says otherwise 😀)

 

Adding the 1/10 option makes confusion as you then must predict what specific module to purchase to match the other module, and given cost is equal, you would rather have the (2off) FTLX1475D3BTL if possible or that Mikrotik cable.

 

So the discussion/speculation/test simply boils down to if those two Mikrotik switches will work with pure 10GB modules when connected to the EtherRegen or any other 1GB device. 
 

If I ever speculate if this will work, it was because it’s possible to force the switch to only be in 1GB mode, and wouldn’t need 1GB hardware (a pure 1GB SFP) in order to obtain this. (Another assumption). Again probably against what Alex have said. And he can still be correct. 
 

The option on general basis is to put a 1GB module in a 10GB switch and/or a 10GB module in a 1GB cage.

You may call that speculation until tested with the Mikrotik or any other 10GB switch if you like. 
 

A $99 switch that comply with the 10GB specification may be a very good way, based on present knowledge, to secure as low jitter and phase noise as possible into the digital chain. Based on the last week knowledge that the FTLX1475D3BTL is working with 1GB devices, it’s of public interest to have these two switches fully tested with that SFP+ module in every possible way against EtherRegen / opticalModule / opticalRendu. 

 

I have no wish or any reason to create any confusion or whatever that will cause uncertainty among readers regarding how to use various SFP’s. I really hope I’m not giving that impression. I’m more interested in the opposite. 

I have said upfront what I like to test. Just to remove the last doubt. And end any possible speculation.

I hope this is clear for everyone. I own all DUT’s except the FTLX1475D3BTL, that need to be purchased. 

 

 

[jabbr]

29 minutes ago, R1200CL said:

We wasn’t, at least I wasn’t, adding 1\10 options into the possibly way of mixing switches and SFP’s

There discussion was using specific two Finisar modules. One pure 1GB (FTLF1421P1BCL) and one pure 10GB (FTLX1475D3BTL)

I did one time suggest to use the 1\10 possibility, but @Superdad clearly stated that wouldn’t help as no auto negotiation would occur. So there was no reason for further discussion in that matter. (Until you added this option again). 

 

 

Uncle!

 

I won't override what @Superdad says about his equipment and what modules are compatible with the EtherRegen. I have certainly used Finisar SFP+ modules in SFP ports successfully. 

 

The Mikrotik cable that has been referenced says that it is 10/1 and presumably does auto-negotiation using the Mikrotik 10G switch. It is also possible that you might need to manually set the port on the switch to 1g. The Intel x520 NIC most certainly auto-negotiates 10g/1g when using the Finisar 10g/1g dual speed SFP+ module (Intel branded).

 

auto-negotiation with SFP+ ports is hit or miss but it does exist and the Finisar 10/1 modules enable that with switches and NICs that also implement. For the Mikrotik I don't remember if you need to manually configure the port or it auto-negotiates. You need to try this yourself and then report results.

 

What @JohnSwenson says about the underlying SFP+ hardware being able to support 1Gbe when in an SFP port is correct (assuming it is a "normal" low powered SFP+ module) but remember that there is also a serial interface and many switches will interrogate the serial interface and then not work with a module it doesn't like for whatever reason. One of the reasons that vendors require "chipped" SFP+ modules is to ensure that only modules that they have tested and meet their own certification are enabled in their switches (its a support headache otherwise). fs.com provides generic SFP+ modules chipped for each vendor at reasonable prices.

 

I know that Sonore is listing "systemOptique" certification for modules but I don't don't know the device actually checks the SFP module to be sure its on the allowed list. 

[R1200CL]

15 minutes ago, jabbr said:

The Mikrotik cable that has been referenced says that it is 10/1

Ops. My bad. 

[ted_b]

Jabbr, et al,

So I brought my "fiber" box upstairs and took inventory.  Do any of these still have a place in a high-end audio network?  I don't want to skimp here, but also not be at least slightly practical.  :)   My first goal is to get signal from my router (home office, first floor) down through the floor to my eventual audio rack on the lower level (not sure my 15M pictured will get me there, but it very much might...but not sure it's the right cabling anyway, at this point.)  I likely need new fiber converters, or an EtherRegen, but as posted earlier, I can't live with 100mbps.

 

Diablo 9 port Gigabit Fiber switch (Model 18198)

Mellanox PCIe card (not pictured, worng PCIe card in pic) MNPA19-XTR ConnectX2 card, a MellanoxConnectx 2 10GbE Ethernet Network Server Adapter MNPA19 XTR

3 TP-Link MC220L FMCs

1 Trendnet TFC-1000MGA FMC

several LC/UPC-LC/UPC Duplex OM3-50/125 fiber cables (2M, 2M, 3M, 10M, 10M and bagged 15M)

sevral SFPs from Cisco and Finsar (Cisco GLC-SX_MM with either FNS or AGS serial numbers, Finsar FTRJ8519P1BNL-B1

 

Any win here is a plus, but not expecting much.  🙂

 

[Superdad]

1 hour ago, jabbr said:

Uncle!

 

LOL, me too!  (I already get dragged down too many rabbit holes in daily life and business...)

 

1 hour ago, jabbr said:

I know that Sonore is listing "systemOptique" certification for modules but I don't don't know the device actually checks the SFP module to be sure its on the allowed list. 

 

I can tell you with certainty that neither the Sonore opticalModule nor the opticalRendu do any SFP code checks. "systemOptique" certification just means they have tested the pieces with their gear and will support it.

[jabbr]

2 hours ago, ted_b said:

 

Jabbr, et al,

So I brought my "fiber" box upstairs and took inventory.  Do any of these still have a place in a high-end audio network?  I don't want to skimp here, but also not be at least slightly practical.  :)  

 

Now we have great options to get fiber all the way to an endpoint. Your fiber and SFP modules are still good.

 

You could get a direct Ethernet DAC ie Lumin

You can get a “NAA” that accepts fiber ie the opticalRendu. There’s also the ClearFog Base than I use as well as the Fitlet 2 which has an SFP option card. 
 

You can get a full optical SFP+ switch eg a Mikrotik or switches with SFP+ ports (many have one or two ports). 
 

You can use an FMC and run your fiber to the endpoint. You can get a opticalModule (Sonore) which is a high end FMC. If you want to use a copper input in your endpoint you can get an EtherRegen, run fiber into it and then copper from the ER to your endpoint. 
 

I would keep the network as simple as possible with as few switches as possible.
 

Basically I like the Roon -> HQPlayer Embedded -> NAA approach but YMMV
 

[ted_b]

Yes, I will most likely have an ethernet dac (exaSound s88 which includes internal NAA).  But unless I turn the EtherRegen around (or you tell me to keep both HQplayer server and exaSound (NAA) ethernet dac on A side...not sure what I'd put on B side) I can't play the larger bitstreamed rates like DSD256 multichannel with 100mbps, so I will have to ponder that.  Is my Diablo switch a no-go?  If so, a Microtik then? 

[jabbr]

16 minutes ago, ted_b said:

Yes, I will most likely have an ethernet dac (exaSound s88 which includes internal NAA).  But unless I turn the EtherRegen around (or you tell me to keep both HQplayer server and exaSound (NAA) ethernet dac on A side...not sure what I'd put on B side) I can't play the larger bitstreamed rates like DSD256 multichannel with 100mbps, so I will have to ponder that.  Is my Diablo switch a no-go?  If so, a Microtik then? 

if you have an ER then you can use the 1Gbe RJ-45, or use the opticalModule to convert fiber to copper, or use the Mikrotik ... get the S-RJ01 copper SFP module to connect out to your DAC ... they even have a 10G S+RJ10 SFP+ module which will connect to your DAC (it will auto negotiate to 1G)

[lmitche]

On 3/18/2021 at 9:55 PM, charlesphoto said:

I got one of those Mikrotik switches once out of curiosity and I can assure you a 1gb SFP works in it. The switch didn't do anything for me so I returned it up the river. 

 

On 3/18/2021 at 9:58 PM, The Computer Audiophile said:

I had high hopes for the Mikrotik I purchased a while ago, but has the same experience as you. 
 

Happy they work great for others though. They are a bargain. 

Did you guys use 1 gbps or 10gbps SFPs(+) with the Microtik switch? Which NIC card or device on the DAC side?

[jabbr]

3 hours ago, lmitche said:

 

Did you guys use 1 gbps or 10gbps SFPs(+) with the Microtik switch? Which NIC card or device on the DAC side?

Both 1 and 10Gbe modules. For the copper modules, some 3rd party ones (probably Brocade) didn’t work, but the Mikrotik branded ones do.

 

I don’t look to a switch to have “SQ” it should be completely silent. The Mikrotik connects to the Arm ClearFog device I have. A Sony PS3 and Intel x520 and Solarflare NICs ... haven’t tested it with Mellanox NICs but it connects with my Mellanox switch just fine. 
 

To reiterate — I’ve never claimed it “sounds better” than any other fiberoptic switch, just  that it’s small, silent and the cheapest SFP+ (10Gbe) that I know of.  SFP+ requires substantially more engineering, attention to board layout, better parts, as well as testing than SFP (1Gbe). That doesn’t mean it necessarily “sounds better”

[charlesphoto]

4 hours ago, lmitche said:

 

Did you guys use 1 gbps or 10gbps SFPs(+) with the Microtik switch? Which NIC card or device on the DAC side?

I don't recall. My reason for returning had nothing to do with sound - I didn't evaluate long enough. It was merely a matter of connectivity - I didn't really need that many SFP ports and my main Cisco 2960 with two is good enough (plus my two opticalModules). IMO this subject is fraught with overthinking. If you really want devices that will 'change' the sound to a degree then best to go with the purpose built etherRegen or opticalModules (or variants from other audiophile mfg). 

[jabbr]

18 minutes ago, charlesphoto said:

Cisco 2960

Hard to argue with the Cisco’s if you don’t mind eBay. 

[R1200CL]

1 hour ago, jabbr said:

I’ve never claimed it “sounds better” than any other fiberoptic switch, just  that it’s small, silent and the cheapest SFP+ (10Gbe) that I know of.  SFP+ requires substantially more engineering, attention to board layout, better parts, as well as testing than SFP (1Gbe).

I think we all have an expectation that due to the fact of lower jitter requirements as well as phase noise, that we ought to obtain the possible best SQ using 10GB equipment that comply with these strict standards. It may be an overkill. 
 

These things is proven by eye stress pattern tests. The only way to document that you comply with the standards. That’s my understanding. I hope I’m correct. 
 

The choice of favor single mode fiber and DFB laser is also based on that this technology is probably best to use.

We could even state that APC connectors instead of UPC should be used. I stick with blue connectors (UPC) and yellow cables.  (That’s most common). That will work for any speed. 

 

The mixing of 1GB devices into this, may be an issue. And maybe not at all. 
My understanding based on John’s answer is that there exist 1GB equipment, such as his own design that have very good implementation in electronics after the cage. Using high quality devices and chips. He also mentioned two technologies that can be used. (There exist only two). As an example, we may assume that’s why some Cisco’s is reported to be very good. Like the 2960G. 
 

I’m expecting, (or speculate if you like), that using 10GB SFP+ modules may be some sort of guarantee that you at least have the technical best option available. And then the best base for god SQ. 
Since SFP28 is very expensive, we can leave those out. Even if they suppose to be technically better. As suggested before, there may be a limit to how far these things shall be stretched. 
 

I’m also wondering how much difference there actually is, technically speaking, between the 1421 and the 1475 modules. I’m not capable of fully understand the minor difference (if any) in the data sheets. I’m not a sure what numbers to look for. But I have several times suggested the jitter numbers. And probably the RIN. Maybe you or @plissken can say something about those two data sheets ? 

I will of cause as several others test some of this. I don’t expect myself to hear much difference. I’m not that good listener. 

 

Power supply used shouldn’t matter when it comes to switches. But obviously it does. Yet to see if this is the matter with a 10GB switch. I have to assume they passed the tests using standard power supplies. 

 

Now I need to go and pick up my opticalModule Deluxe. Just got SMS from DHL 😀
 

 

finisar_ftlx1475d3btl_rohs-6_compliant_10g_10km_1310nm_single_mode_datacom_sfp_transceiver_productspecrevb1_0.pdf finisar_ftlf1421p1xtl_oc-48_ir-1_stm_s-16.1_rohs_compliant_pluggable_sfp_transceiver_product_specification_rev_b1.pdf

[jabbr]

3 hours ago, R1200CL said:

I think we all have an expectation that due to the fact of lower jitter requirements as well as phase noise, that we ought to obtain the possible best SQ using 10GB equipment that comply with these strict standards. It may be an overkill. 
 

These things is proven by eye stress pattern tests. The only way to document that you comply with the standards. That’s my understanding. I hope I’m correct. 
 

The choice of favor single mode fiber and DFB laser is also based on that this technology is probably best to use.

We could even state that APC connectors instead of UPC should be used. I stick with blue connectors (UPC) and yellow cables.  (That’s most common). That will work for any speed. 

 

 

This thinking is not wrong. Its easy to do. 

 

I use the Finisar FTLX1475 this is an SFP+ single mode module, it comes in two versions:

FTLX1475D3BCV ... the "D3BCV" version is the dual 10G/1G module for both 10GBase-LR and 1000base-LX

FTLX1475D3BTL ... the "D3BTL" version is the 10G module for 10GBase-LR

 

electrically as @JohnSwenson has discussed you can use these modules in an SFP port and they work. The caveat is that Finisar states that they are not for use in an SFP port but I have tested this to work in the Clearfog Base, and apparently he has tested this to work in the EtherREGEN. YMMV.

 

So what is the difference between the two versions of the FTLX1475?

 

The dual rate "D3BCV" version uses the RS0 line (rate select). The rate select lines interact with the switch. According to the datasheet:

Transceiver data rate selected through the 2-wire bus in accordance with SFF-8472 Rev. 10.3. Soft RS0 is set at
Bit3, Byte 110, Address A2h. Soft RS0 default state on power up is ‘0’ LOW, and the state is reset following a
power cycle. Writing ‘1’ HIGH selects max. data rate operation. Transceiver data rate is the logic OR of the input
state of the RS0 pin and soft RS0 bit. Thus, if either the RS0 pin OR the soft RS0 bit is HIGH then the selected
data rate will be 9.95 and 10.3 Gb/s. Conversely, to select data rate 1.25 Gb/s both the RS0 pin and the soft RS0
bit are set LOW.

The single rate "D3BTL" version does not use the RS0 line and works at 10GBase. 

 

The equivalent multimode FTLX8574 also uses the same extensions to denote the 10G and dual 10G/1G versions. The 8574 is the module I reported on in the first post of this thread and yes at that time this SFP+ module worked in an SFP port (I suspect this was dumb luck on my part rather than some grand design).

[stray_cat]

That's how my thinking was when I went optical. It made sense to me that while 10gbe would really be overkill (my service level is only 200-250 mbps) and the ER feeds the streamer at only 100mbps (read it somewhere), the requirements/specs of the 10gbps parts would put that section of my chain ahead of the ER & my streamer above reproach and allow me to rule out that section as a source of degradation.

My chain is stock Xfinity wireless gateway router->AQ cinnamon->mikrotik CRS305-1G-45+IN->Finisar FTLX1471D3BCV(x2)->Mikrotik CRS305->Finisar FTLF1421P1BTL(x2)->EtherRegen(w/LPS 1.2)SFP port->Sablon Panatela->dcsNB streamer.

 

I'll have to say that adding the 10gbps mikrotik w/Finisars section brought qobuz/Tidal streaming up a couple levels of SQ even before installing the ER & feeding it via its SFP port from the 2nd Mikrotik. If there was any added SQ from the upgrades past the x2 Mikrotik section, they were likely less significant, to these ears.

 

Adding an Afterdark queen clock next to the ER to see if better SQ can be gained.

[ericuco]

In my setup, I have an Intel X520-DA1 NIC (10 Gbe) installed in my music server that is directly connected to my Mikrotik switch. The switch also connects to my opticalRendu. So two fiber connections to switch. I also use the Ethernet port to connect to rest of my network including Roon server, NAS, Internet, etc.

 

At one point I was using SFP+ (10 Gbe) Finisar modules in my music server to switch connection but I noticed that I was getting a huge number (more than 10,000, maybe 100,000 ... in any case, a constant stream of "errors") pause requests when feeding the SFP (1 Gbe) modules for the oR. I switched from the SFP+ modules to SFP modules in my music server and am now not seeing many "errors" at all as noted below.

 

My thinking is that the Mikrotik switch is having to work "harder" with all the pause requests ("errors") when using the SFP+ modules. Using SFP modules for everything, I get very few "errors" as noted below. Generally speaking, the goal of all related computers (including switches) is to reduce any unnecessary activity is the logic I used ... no idea if this is valid or not. There was little if any change in SQ that I detected. In any case, something to consider.

 

 

[MarkusBarkus]

Nice/helpful post @ericucoThank you!

[jabbr]

8 hours ago, ericuco said:

In my setup, I have an Intel X520-DA1 NIC (10 Gbe) installed in my music server that is directly connected to my Mikrotik switch. The switch also connects to my opticalRendu. So two fiber connections to switch. I also use the Ethernet port to connect to rest of my network including Roon server, NAS, Internet, etc.

 

At one point I was using SFP+ (10 Gbe) Finisar modules in my music server to switch connection but I noticed that I was getting a huge number (more than 10,000, maybe 100,000 ... in any case, a constant stream of "errors") pause requests when feeding the SFP (1 Gbe) modules for the oR. I switched from the SFP+ modules to SFP modules in my music server and am now not seeing many "errors" at all as noted below.

 

My thinking is that the Mikrotik switch is having to work "harder" with all the pause requests ("errors") when using the SFP+ modules. Using SFP modules for everything, I get very few "errors" as noted below. Generally speaking, the goal of all related computers (including switches) is to reduce any unnecessary activity is the logic I used ... no idea if this is valid or not. There was little if any change in SQ that I detected. In any case, something to consider.

 

 

I’ve heard the device can’t handle a full 1Gbe and issues lots of pause frames. What happens when you set the Mikrotik port to 1Gbe? I don’t think pause frames are a problem though, they are normal behavior.

[Duke40]

I am also getting a large number of pause frames with 1Gbe, just reset the error count and stats below for about 1 song for RX pauses.

 

Does not worry me, actually I would be surprised if no pause frames happened, afterall my SFPs are 1Gbe which connect to EtherRegens which have a 100Mb output on the "B" side.

 

I kind of expect to have RX pauses, maybe its even a good thing, may provide yet another buffer between the Nucleus and ER.

Maybe it does not matter at all (2nd screenshot shows just how little traffic happens in my network).

 

While these pause frames are shown in the Errors page on SwitchOS, I dont consider them an actual error, more just informational.

Plus, apart from "RX pauses", I have never seen any errors ever with this switch, not once.

 

1st screenshot is of the Errors page.

2nd screenshot is of Stats page ... this is about as busy as MikroTik CRS305 switch ever gets (one stream of music, one of netflix, plus using my laptop).

 

[R1200CL]

Maybe there is a way to avoid them ? @plissken This your domain maybe?

[Duke40]

Also, I did play around a bit with different settings on the MikroTik CRS305.

Had a quick check with setting up VLANs, did not hear any difference, or see any difference for errors/pauses and such.

 

Though I did give "Port Isolation" a go, preventing my Roon Nucleus and Apple TV4K from seeing each other as there is no need, that seemed to give a very subtle sound uptick. For perspective, powering the switch with an excellent LPS provided more noticeable difference. Also need to be careful with "Port Isolation" so I do not accidentally turn off too much access and get trouble accessing the switch.

 

Will give 10Gbe cables a go once Mikrotik S+AO0005 cables arrive (in about 4 weeks).

 

This MikroTik CRS305 setup has been a fun and interesting experiment, which has also provided me electrical isolation from my AV setup, as this is a goal of mine. I am doing a Network Switching subject later this year at online university, so its great to have something setup ready for learning later on with the MikroTik.  Plus if I can get an improvement for sound/video as part of optimising my home gym/office AV network, that is great [though I think I may have achieved that by simply installing the CRS305 switch with a great LPS, just need to finalise a choice on the best SFP/cables to use, for which I hope the Mikrotik S+AO0005 delivers].

[R1200CL]

@Duke40

Have you enable Flow control ? (Under Link). 

https://en.wikipedia.org/wiki/Ethernet_flow_control