Optical Network Configurations

[Bertel]

I have a question regarding IEEE 802.3x Flow Control on the Mikrotik CRS305-1G-4S+ switch, maybe one of the knowledgeable guys here has a hint (especially looking at you @jabbr 😉):

 

In SwOS I have enabled both Tx and Rx for all four SFP+ ports. While the 10G connections are fine, both 1G connections say 'rx only' on the 'Link' tab (pls see screenshot - SFP1 is my SignatureRendu, SFP2 is my internet router, SFP3 is my NAS, and SFP4 is my HQPlayer server). Everything works fine most of the time, but occasionally there's a 1-3 second pause until playback resumes (rarely, maybe once or twice per listening session, also irregularly and independent of high/low bit rates or PCM/DSD files). On the 'Errors' tab I see a massive number for Tx Pauses, a moderate number for Rx Pauses, but a disturbing number for actual Rx Overruns (See second screenshot). I assume this points to the root cause for the occasional dropouts, but struggle to make sense of actually where to look and what to fix... Any ideas? 

[jabbr]

4 hours ago, audiobomber said:

The Finisar FTLF1318P3BT SFP's that @tdx currently owns definitely work with the EtherRegen. I use the same SFP in my opticalModule Deluxe and ER.

Folks are missing the point. Its not that the module doesn't work in the EtherRegen ... its that the module may not work with the Mikrotik ... there is no presumption that an optical link use the same SFP modules on both ends: which is why I suggested that he use the SFP(+) module on the Mikrotik that I use (happens to be a dual 10g/1g)

[jabbr]

1 hour ago, Bertel said:

pls see screenshot - SFP1 is my SignatureRendu, SFP2 is my internet router, SFP3 is my NAS, and SFP4 is my HQPlayer server

This doesn't make sense to me because the SFP1 and SFP2 show 10g links but I'm assuming they aren't... its possible that the SignatureRendu, which IIRC can't handle full 1g is throttling the switch (as it should).

[jabbr]

2 hours ago, lasker98 said:

I use the FTLF1318P3BT SFPs with ER and a Mikrotik css610 switch with no problems.

That's great feedback ... do you set the CSS610 port to 1g or autonegotiate?

[89reksal]

25 minutes ago, jabbr said:

do you set the CSS610 port to 1g or autonegotiate?

I have it set for autonegotiate. In the pic below, SFP1 (DAC) goes to Etheregen then DAC/streamer. SFP2 (ST) goes to Optical Module Deluxe then Sonictransporter i5

[vortecjr]

2 minutes ago, lasker98 said:

I have it set for autonegotiate.

I have no expierence with this device, but I would check off Flow Control as well. 

[89reksal]

2 minutes ago, vortecjr said:

I have no expierence with this device, but I would check off Flow Control as well. 

Thanks Jesus. I thought so as well but having Flow Control enabled I get errors (RX and TX pauses). Not noticeably audible but they show up on Errors page.

[vortecjr]

Go to the Rendu's home page and add /diag to the units IP address and post the content of that page here.

[Bertel]

1 hour ago, jabbr said:

This doesn't make sense to me because the SFP1 and SFP2 show 10g links but I'm assuming they aren't... its possible that the SignatureRendu, which IIRC can't handle full 1g is throttling the switch (as it should).

 

Oh sorry, you're right, I got the list wrong - it's exactly the other way round: SFP1 is my HQPlayer server (the NIC is set to 10G), SFP2 is my NAS (it can actually do 25G but is set to 10G as well), SFP3 is my internet router, and SFP4 is my SignatureRendu.

 

And yes, I think I remember having read that a few days ago as well - just found it, @Miska wrote yesterday

"Remember that the Rendu can do max 400 Mbps in ideal conditions (flow control functional). This is bandwidth limitation of the internal bus between the ethernet controller and CPU. If flow control is not functional, it will have occasional stalls where it drops to practically 0 for some duration. This is because the ethernet controller's packet buffer overflows without flow control. Which in turn causes packet loss, which in turn causes re-sends, which in turn causes more overflow/packet loss and so on. This causes complete traffic jam." (here: https://audiophilestyle.com/forums/topic/19715-hq-player/?do=findComment&comment=1237263 )

 

Absolutely, SignatureRendu via flow control should throttle the switch - from my layman's understanding I thought that this is where the TX and RX Pauses com from, which I thought is fine - but I do have the occasional 1-3 second pauses (where I think HQPlayer has to stop playing, I can hear the fans go down until a few seconds playback resumes and the fans spin up again), I thought the Rx Overrun might correlate with that...?

 

 

[vortecjr]

Just know that the opticalRendu have an on board switch and flow control is requested of it on boot. This can be confirmed from the page I pointed you to above.

[Bertel]

9 minutes ago, vortecjr said:

Just know that the opticalRendu have an on board switch and flow control is requested of it on boot. This can be confirmed from the page I pointed you to above.

 

Sorry Jesus, had missed that - here are the Network Statistics and Network Pause Settings sections of the System Diagnostic report:

 

Network Statistics

Kernel Interface table
eth0: flags=4163  mtu 1500
        inet 192.168.178.80  netmask 255.255.255.0  broadcast 192.168.178.255
        inet6 fe80::d263:b4ff:fe02:fc25  prefixlen 64  scopeid 0x20
        ether d0:63:b4:02:fc:25  txqueuelen 1000  (Ethernet)
        RX packets 29257390  bytes 3175638082 (2.9 GiB)
        RX errors 691  dropped 0  overruns 0  frame 691
        TX packets 838496  bytes 63621025 (60.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 636  bytes 72956 (71.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 636  bytes 72956 (71.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

NIC statistics:
     tx_dropped: 0
     tx_packets: 52168
     tx_broadcast: 9177
     tx_multicast: 51011
     tx_crc_errors: 0
     tx_undersize: 0
     tx_oversize: 0
     tx_fragment: 0
     tx_jabber: 0
     tx_collision: 0
     tx_64byte: 49879
     tx_65to127byte: 61920
     tx_128to255byte: 3435
     tx_256to511byte: 2204
     tx_512to1023byte: 27
     tx_1024to2047byte: 239
     tx_GTE2048byte: 0
     tx_octets: 2327498921
     IEEE_tx_drop: 0
     IEEE_tx_frame_ok: 52168
     IEEE_tx_1col: 0
     IEEE_tx_mcol: 0
     IEEE_tx_def: 0
     IEEE_tx_lcol: 0
     IEEE_tx_excol: 0
     IEEE_tx_macerr: 0
     IEEE_tx_cserr: 0
     IEEE_tx_sqe: 0
     IEEE_tx_fdxfc: 46170
     IEEE_tx_octets_ok: 2327498921
     rx_packets: 24569
     rx_broadcast: 35276
     rx_multicast: 58170
     rx_crc_errors: 0
     rx_undersize: 0
     rx_oversize: 0
     rx_fragment: 0
     rx_jabber: 0
     rx_64byte: 15840
     rx_65to127byte: 48694
     rx_128to255byte: 25269
     rx_256to511byte: 57068
     rx_512to1023byte: 59665
     rx_1024to2047byte: 14641
     rx_GTE2048byte: 0
     rx_octets: 1134963392
     IEEE_rx_drop: 0
     IEEE_rx_frame_ok: 24569
     IEEE_rx_crc: 0
     IEEE_rx_align: 0
     IEEE_rx_macerr: 0
     IEEE_rx_fdxfc: 0
     IEEE_rx_octets_ok: 1134963392

 

Network Pause Settings

Pause parameters for eth0:
Autonegotiate:	on
RX:		on
TX:		on
RX negotiated:	on
TX negotiated:	on

Pause parameters for eth1:

 

Looks like the settings are 'on' so ok?

 

I had read somewhere that some switches revert to 'rx only' with bonded interfaces, could that be the case here? But would that actually cause the stops...?

 

[vortecjr]

1 hour ago, Bertel said:

Looks like the settings are 'on' so ok?

 

I had read somewhere that some switches revert to 'rx only' with bonded interfaces, could that be the case here? But would that actually cause the stops...?

 

The RX errors are not normal. I’m not certain about the bonded interface. 

[jabbr]

5 hours ago, lasker98 said:

Thanks Jesus. I thought so as well but having Flow Control enabled I get errors (RX and TX pauses). Not noticeably audible but they show up on Errors page.

I’m not sure that these are errors? Perhaps requested pauses … @Miska is much more knowledgeable on this than me 🤷🏻‍♂️ 

 

If they aren’t errors then when going from 10g to 1g it makes sense that the switch might tell the 10g device to slow down ie throttle 

[89reksal]

12 hours ago, jabbr said:

I’m not sure that these are errors? Perhaps requested pauses …

That's an interesting thought. I just assumed since they showed up on the errors page they were "bad". I'd be very interested in hearing any more thoughts on this. It does make sense.

[tdx]

22 hours ago, lasker98 said:

I wonder if it would be worth buying or borrowing a cheap FMC to try in place of the ER. I'm definitely no expert but other than your fiber cable being defective (seems unlikely but cheap enough to order another with the FMC) or the SFP not inserted all the way into the ER, it looks like the ER is the only thing that hasn't been replaced.

I use the FTLF1318P3BT SFPs with ER and a Mikrotik css610 switch with no problems. I'm assuming you power down the ER when installing the SFP? I do find in my system it's necessary to get the SFP recognized. I also power down my dac/streamer at same time. After inserting the SFP in the ER, power up the ER and check for activity lights. Then power on the dac streamer.

I also strongly recommend an app called Net Analyzer. It will show everything active on the network.

It just occurred to me as well; why not connect your two switches in series using fiber between them and then connect the last switch with ethernet to your ER? This will eliminate need to buy an FMC at all and prove if all your fiber components are working properly.

 

So I spent all morning trying various combinations of the Mikrotik and the QNAP switch together, and it looks like the issue is with the SFP module and/or the patch cable. To further narrow it down i went ahead and ordered a different patch cable that was recommended in this thread, as well as an SFP module that is officially compatible with the QNAP switch (which I will end up keeping, as I like it better thank the Mikrotik anyway)

 

I will try more combos and hopefully can isolate the culprit. Thank god for Amazon's lenient return policy :)

[audiobomber]

52 minutes ago, tdx said:

QNAP switch (which I will end up keeping, as I like it better than the Mikrotik anyway)

Based on sound quality? Build quality? Cost? Something else?

[Miska]

20 hours ago, Bertel said:

Absolutely, SignatureRendu via flow control should throttle the switch - from my layman's understanding I thought that this is where the TX and RX Pauses com from, which I thought is fine - but I do have the occasional 1-3 second pauses (where I think HQPlayer has to stop playing, I can hear the fans go down until a few seconds playback resumes and the fans spin up again), I thought the Rx Overrun might correlate with that...?

 

If those pauses appear not because of CPU loading, but instead because of network, it is most likely due to network stalls caused by non-functional flow control.

 

20 hours ago, Bertel said:

Looks like the settings are 'on' so ok?

 

On that link, but please remember it needs to be properly propagated throughout network. So for each "cable step" aka link on the network between the final endpoints.

 

[Raypok]

I just tested and can confirm that no problem in connection between Mikrotik CSS610 to MIkrotik CRS305-1G-4S+ using either Finisar FTLX1475D3BCL both end & Finisar FTLF1318P3BTL both end. CSS610 set autonegotiate by default.

[tdx]

1 hour ago, audiobomber said:

Based on sound quality? Build quality? Cost? Something else?

Two things really: the QNAP has a mix of ports incl. 10GBE RJ45 and SFP which is useful to me. Also, and this is just me, the Mikrotik has a very bright blue glow that didnt work in my space. Otherwise it's a great unit.

[tdx]

32 minutes ago, Raypok said:

I just tested and can confirm that no problem in connection between Mikrotik CSS610 to MIkrotik CRS305-1G-4S+ using either Finisar FTLX1475D3BCL both end & Finisar FTLF1318P3BTL both end. CSS610 set autonegotiate by default.

thanks so much for testing this. Can I ask you the specific patch cable you used?

[R1200CL]

40 minutes ago, Raypok said:

Finisar FTLX1475D3BCL both end

I don’t think that will work with Etherregen or OpticalRendu. I know 1475 + 1421 works together. This is because same laser as well as one module is 1GB, and the other 10Gb. 1GB must be in Mikrotik. 

[Raypok]

I am using corning 3 meter fiber patch cord with  LC LC connector Single mode 9/125um

[Raypok]

I haven't tested with ER yet, only between both Mikrotik

[tdx]

Tried a new/different patch cable, still no signal. So it really looks like one or both of the SFP modules are either incompatible or defective. I'll wait for the new SFP module to test and hope it fixes it.

[R1200CL]

@Raypok. @tdx

You are using a yellow cable with blue connectors? 
If you have tried anything else, you may have destroyed the SFP interface.