Optical Network Configurations

[89reksal]

I wonder if it would be worth buying or borrowing a cheap FMC to try in place of the ER. I'm definitely no expert but other than your fiber cable being defective (seems unlikely but cheap enough to order another with the FMC) or the SFP not inserted all the way into the ER, it looks like the ER is the only thing that hasn't been replaced.

I use the FTLF1318P3BT SFPs with ER and a Mikrotik css610 switch with no problems. I'm assuming you power down the ER when installing the SFP? I do find in my system it's necessary to get the SFP recognized. I also power down my dac/streamer at same time. After inserting the SFP in the ER, power up the ER and check for activity lights. Then power on the dac streamer.

I also strongly recommend an app called Net Analyzer. It will show everything active on the network.

It just occurred to me as well; why not connect your two switches in series using fiber between them and then connect the last switch with ethernet to your ER? This will eliminate need to buy an FMC at all and prove if all your fiber components are working properly.

 

[89reksal]

25 minutes ago, jabbr said:

do you set the CSS610 port to 1g or autonegotiate?

I have it set for autonegotiate. In the pic below, SFP1 (DAC) goes to Etheregen then DAC/streamer. SFP2 (ST) goes to Optical Module Deluxe then Sonictransporter i5

[89reksal]

2 minutes ago, vortecjr said:

I have no expierence with this device, but I would check off Flow Control as well. 

Thanks Jesus. I thought so as well but having Flow Control enabled I get errors (RX and TX pauses). Not noticeably audible but they show up on Errors page.

[89reksal]

12 hours ago, jabbr said:

I’m not sure that these are errors? Perhaps requested pauses …

That's an interesting thought. I just assumed since they showed up on the errors page they were "bad". I'd be very interested in hearing any more thoughts on this. It does make sense.

[89reksal]

I'm trying to get SFP+ Finisar FTLX1471D3BCL modules working between my Mikrotik CRS305 switch and ethergen (ER). I was previously using Finisar SFP modules FTLF1318P3BTL with no issues between Mikrotik CSS610 and ER (as well as optical module deluxe). The ER connects to my Bryston dac/streamer and the OM Deluxe connects to my Sonictransporter.

 

I'm currently only able to get connections with the FTLX1471D3BCL SFP+ modules installed in ER and OM Deluxe and FTLF1318P3BTL SFP  modules installed in either switch. I originally was trying FTLX1471D3BCL SFP+ modules with the Mikrotik CSS610 switch and found no way to get them to connect. I'm aware that the switch has to be operating at 1 GB but it seems the CSS610 is unable to be manually set for 1GB using the FTLX1471D3BCL SFP+ modules. This seems to be a known issue with the CSS610 switch.

 

After more reading, (especially this post by Jud)

 

I decided to buy the Mikrotik CRS305. I installed the SFP+ modules and was very surprised to be unable to make a connection to the ER and OM Deluxe. I upgraded the firmware to 2.14 (was 2.13) but no change. I tried manually setting 1GB link speeds but nothing. Even if I manually set 1GB while using the FTLF1318P3BTL SFP modules I lose connection. Only auto negotiate works.

 

If anyone (Jud?) that has one of these Mikrotik switches working with SFP+ modules at both the switch and ER or OM Deluxe ends, I'd very much appreciate any ideas or suggestions. I see on the After Dark forums, it seems SFP+ is working on both ends using various switches and even cheap FMCs. I'm hoping it's just some simple setting I'm missing. I'm using SwOS in CRS305.

[89reksal]

21 hours ago, R1200CL said:

Choosing the 1GB in settings won’t work.

This has been my experience as well with both the  Mikrotik CSS610 and CRS305. 

 

14 hours ago, Jud said:

I use these at both the Mikrotik and ER ends.

Those are SFP not SFP+. You were my last hope since your original post was the only example I've found where someone specifically said they had SFP+ working installed at each end of CRS305 and ER.

 

13 hours ago, jabbr said:

... is that the 10g/1g SFP(+) modules work on the Mikrotik side.

I may try this at some point but for now I guess I'll have to be satisfied with SFP at switch and SFP+ at ER and OM.

[89reksal]

1 hour ago, Jud said:

SFP+ at switch and SFP at ER?

Not sure what you're referring to here? My current configuration is with Finisar FTLF1318P3BTL SFP modules at switch and Finisar FTLX1471D3BCL SFP+ modules at ER and OM Deluxe. The link speeds are 1GB with auto negotiation enabled in the switch (CRS305).

 

In my original post I mistakenly stated I updated the switch firmware from 2.13 to 2.14. This should have read 2.12 to 2.13. 2.13 is the latest version for SwOS as far as I know.

 

 

[89reksal]

1 hour ago, Jud said:

I just thought it was interesting that you had an SFP+ working on the 1gbps ER port and an SFP on the 10gbps Mikrotik port.

Ah, I see what you meant now. Yes, definitely interesting, especially with ER being only 100 Mbps yet it can negotiate with the SFP+ module but the 10 GB Mikrotik switch can't? Even when manually set for 1 GB? I have to believe there's some kind of issue with the Mikrotik switch.

[89reksal]

1 hour ago, Superdad said:

Optical does not do speed autonegotiation. What happens is here is a pin on the SFP and SFP+ modules that is used by the switch port to determine the speed. An SFP module grounds the pin. An SFP+ leaves it floating.

Thanks Superdad. I do remember reading similar explanation from John Swenson on how they worked. I actually have it saved and was using it as a reference while trying to work through all this. I even came across this setting in the Mikrotik CRS305 RouterOS interface settings:  

 

sfp-rate-select (high | low; Default: high)

 

I couldn't find any real explanation of what exactly it does but I was hoping it would allow me to manually configure how the control rate select pin for SFP+ ports works just as you described. I tried both high and low setting but no difference so not sure what it's doing.

1 hour ago, Superdad said:

The upshot is the SFP+ on both ends, the ports must also be SFP. If one port is SFP and the other SFP+ one end will run at gig and the other at 10 gig and no communication is done. So if you wave a switch with 10G port and ER (which is 1G) you must use SFP modules on both ends.

Based on what you're saying, the CRS305, which is 10GB, with SFP module installed and the ER, which is 1GB, with SFP+ module installed, shouldn't be able to communicate (assuming I'm understanding what you're saying) yet that's exactly the configuration I have working. Something definitely seems odd.

[89reksal]

7 hours ago, Superdad said:

 

Deleted

[89reksal]

2 hours ago, pl_svn said:

please... do these Finisar FTLF1318P3BTL look legit ones?

No idea how to tell legit from fake but yours look ok compared to mine. I bought cheap from an ebay seller in China. 

[89reksal]

6 hours ago, jabbr said:

People can't just do random stuff and complain when it doesn't work.

Mikrotik CRS305 and CSS610 are both 10Gb switches that have SFP+ ports. Both have settings for auto negotiation and manual link speed (10/100/1G/10G) selection. Unless I completely misunderstand how those settings should operate, I expect manually selecting 1G when using an SFP+ module in one of these switches to be able to connect to a device operating at 1G, whether the device is using SFP or SFP+ module.

 

There has been no reported cases of either of these switches being able to do this. I am also speaking from personal experience since I have both these switches. Seems obvious any complaints are valid. 

[89reksal]

2 hours ago, jabbr said:

I’ve posted several times exactly how to use an SFP(+) module with the Mikrotik at 1Gbe. This is why God created the dual 10G/1G modules.

What you (and others)have posted is how to use a dual rate 1G/10G SFP+ in Mikrotik at 1G. That's a workaround, not a solution to using standard SFP+. If that's the only solution, then why did God create managed switches, such as the Mikrotiks we're discussing, with auto negotiation and option to manually select link speeds from 10M/100M/1G? There would be no need for dual rate SFP+ in 10G switches running ports at 1G if this was working correctly. In the Mikrotik switches we're discussing, it does not work correctly. The sooner people stop posting that it does, than the sooner people (such as myself) can stop wasting money buying these switches and modules under the assumption that they will work with SFP+ modules at 1G.

 

With all due respect jabbr, you're the reason I initially switched to optical fiber. Everything worked great with my SFP modules in my Mikrotik CSS610 into SFP modules in my ER and Optical Module Deluxe at 1G. It was plug and play and I was a fiber convert. You are also the reason I decided to explore 10G with SFP+ modules, based on your posts about the benefits of 10G. So I do respect your opinions.

 

Have you actually tried either of these switches? Did you try the various software settings and find they did or didn't work as expected? I can only assume you haven't otherwise I have no other explanation for your comments. Even John Swenson, someone else whose opinions I respect immensely, posted this when discussing SFP+: "If you use a switch with 10GbE ports that can be switched to 1GbE, they will work fine, but If the port ONLY runs at 10GbE, you cannot use it with a link where the other end is SFP." Link to entire post:

He also posted a good explanation of dual rate SFP+ modules if anyone is interested:

 

[89reksal]

13 hours ago, Bertel said:

15 hours ago, lasker98 said:

Mikrotik CRS305 and CSS610 are both 10Gb switches that have SFP+ ports. Both have settings for auto negotiation and manual link speed (10/100/1G/10G) selection. Unless I completely misunderstand how those settings should operate, I expect manually selecting 1G when using an SFP+ module in one of these switches to be able to connect to a device operating at 1G, whether the device is using SFP or SFP+ module.

 

Correct, that's exactly how my CRS305 works here

To clarify, you're stating that you have installed a non-dual rate SFP+ in one of the CRS305 SFP+ ports and have it communicate with a 1G device that has an SFP+ installed? If yes, you're the first confirmed case of someone successfully implementing this configuration. Please show screen shots of all relevant settings you've made in your Mikrotik OS, also showing the exact module used as shown in the OS. Hopefully this will allow myself and others to duplicate your success.

[89reksal]

3 hours ago, jabbr said:

I have a Mikrotik 10g switch and use it to connect to my 1g endpoints. I have been since the very day I installed it which was before I posted that it works...

I'm assuming you're again referring to implementing this using dual rate SFP+? Will you confirm that you believe your Mikrotik switch (CRS305 or CSS610) is working correctly even though it doesn't respond to the auto negotiation or manual link speed settings in the switch OS when using non-dual rate SFP+ in your switch when connecting to your 1G endpoints?

 

I'm trying to be as specific as I can with my questions in hopes of getting an equally specific reply.

[89reksal]

2 hours ago, Jud said:

(1) Whether a sending SFP+ module/switch can be set to only send SFP rates;

 

2 hours ago, Jud said:

But since we have less experience than John, for whom the distinction is probably implicit, it was easy for us to take this (incorrectly) as being true of context #1 as well.

I think you're trying to say (1) above is incorrect? If yes, I have to disagree. Based on what John actually posted:

 

"So in summary,

all SFP+ modules will work fine in SFP cages as long as you don't mix speeds of the devices, both ends must be running at gigabit. If you use a switch with 10GbE ports that can be switched to 1GbE, they will work fine, but If the port ONLY runs at 10GbE, you cannot use it with a link where the other end is SFP." 

 

Pretty clear that if switch can be set for 1G link speed, which both CRS305 and CSS610 have settings for in their OS, then based on what John actually said, "they will work fine" (where "they" is referring to SFP+ modules). For whatever reason, neither of these Mikrotik switches do work as John describes. I don't understand why people want to keep trying to say they are working correctly. If this is correct operation, please explain the correct purpose of the 10/100/1G manual link speed setting if it's purpose is not to manually set link speed to 10/100/1G. 

 

[89reksal]

12 hours ago, jabbr said:

yes it’s working correctly.

Thanks. I still haven't been able to get an explanation (after asking multiple times) of what the speed setting in the "Link" tab of the SwOS user interface, which has option to select from 10M/100M/1G, is supposed to do if it's purpose is not to set the link speed of the selected port to 10M, 100M or 1G. I have to assume God created that option for some reason. Hopefully you can finally answer this for me.

[89reksal]

2 hours ago, Jud said:

what SwOS version?

2.13

 

2 hours ago, jabbr said:

I set the link speed to what I want it to be.

Ok, I'll bite; how do you do that since the speed select function doesn't work?

[89reksal]

4 hours ago, jabbr said:

Are you using a dual rate module?

In my Mikrotik CRS305 I had two Finisar FTLX1475D3BCV dual rate modules working for two days last week. I purchased these new from Mouser. I also purchased two new FTLX1475D3BCL SFP+ modules at the same time. I purchased these mainly based on your and others experience and comments that these dual rate modules were able to be used in this switch as a confirmed configuration to have an SFP+ module operating at 1G. 

 

Even these were not simple to get working initially. These replaced two Finisar FTLF1318P3BTL SFP modules that were operating at 1G with "auto negotiation" selected in SwOS ver. 2.13. On the other end of these two links were two FTLX1471D3BCL SFP+ modules, one installed in my ER, the second installed in my Optical Module Deluxe. This configuration was rock solid. It always consistently worked after all failed attempts trying different SFP+ modules in the Mikrotik. It was just plug and play.

 

Getting back to this part, "not simple to get working initially". Based on previous comments made about the 1475 dual rate modules, I fully expected to simply exchange the 1318 modules for the new 1475 dual rate modules, still with "auto negotiation" selected. Nothing, no link established. The new modules were correctly recognized and identified in the SwOS "SFP" tab. I tried manually setting the link speed to 1G and still nothing. All these attempts also involved various combinations of power downs, reboots, etc., but nothing. I thought maybe it was because I was still using the 1471 modules at the other end. I replaced these with the new FTLX1475D3BCL modules, still nothing (again after trying multiple power down, reboot, etc. combinations).

 

Back at the Mikrotik end, after multiple attempts, even trying to set up in RouterOS, I somehow got 1G links established after trying all the different manual link speed options back in SwOS. The last thing I did was manually set 100M, then set "auto negotiation". Everything worked. This all took well over an hour and about $500.00 (Canadian) for the four new modules. Not thrilled but at least I finally had a 10G switch working with SFP+ modules.

 

Two days later I went to play music and nothing. After about an hour of trying everything I had tried to get it working originally, I went back to the 1318 modules and again, these instantly worked. As a side note, thank God for the Net Analyzer app. I can't imagine going through all this without it.

 

So to finally answer your original question;

4 hours ago, jabbr said:

Are you using a dual rate module?

no. If someone is able to tell me how to get them working in my switch at 1G it would be much appreciated.

[89reksal]

11 minutes ago, Jud said:

You realize neither the article you cited about vendor lock-in nor the very first post under the John Swenson search support what you are saying?

I assume you're referring to R1200CL's response to this post "Again the SFP module has to both be supported by the switch and support the selected speed. It’s data sheet states which speeds it supports." by jabbr?

 

This is what comes up as "the very first post under the John Swenson search":

 

"Just some more info on SFP+ ports and modules. As far as I know ALL switches with SFP+ ports will automatically run the port at 1Gb when an SFP module is plugged it. There are a few switches (they HAVE to be managed switches) let you program the port to a fixed 1Gb , so it will run at 1Gb even with an SFP+ module plugged in. The opposite does not make sense since an SFP module will not run at 10Gb. BUT an SFP+ module WILL run at 1Gb! So you CAN actually put an SFP+ module into an ER! So IF you have one of those switches that lets you program the speed of a port you could actually use SFP+ modules at both ends.

 

 Note that for all of these switches that let you program the port speed, the default behavior is to look at the module type to determine the speed. If you want to use an SFP+ module with an SFP+ port with an ER you HAVE to have a switch that supports manual speed selection and you HAVE to go in an program it. You have to be comfortable with programming your switch."

 

I'd say that couldn't be clearer in supporting what R1200CL is saying. What it doesn't support is what jabbr is saying.

[89reksal]

21 minutes ago, dbastin said:

He suggested ... untick the 10G 'advertisement' ..  perhaps you know what this means in case you haven't tried it (I havent).

It's accessed through RouterOS. Use "WebFig" (selected at top right of RouterOS UI (you'll see "Quick Set" "WebFig" "Terminal" options). Then from the left side menu selections, choose "Interfaces", then you'll see your ports. Under "Name" tab, click on the port you want. That will open a new page showing link options for that port. Scroll down and you will see "Advertise" with all the different options. On mine, "10G full" is already unchecked for all ports.

 

Be aware that as far as I know, settings made in RouterOS are only in effect while in RouterOS. They have no effect once back in SwOS. If I'm wrong on this anyone please feel free to correct me.

[89reksal]

27 minutes ago, Jud said:

"Program" doesn't imply to me set and forget in a GUI.

I read it in a much more generic way as John used it. I take it to simply mean it has to be set somehow through the switch management console. Not all switches have a more simple GUI like  SwOS in our Mikrotik switches, that allows you to manage some settings (such as link speed which apparently doesn't function correctly in this use case) without getting into more advanced configuration through command line or RouterOS. My two Mikrotik switches are the only managed switches I have any experience with.

[89reksal]

3 hours ago, Bertel said:

what usually works for me is to untick the 'Auto Negotiation' box on the 'Link' tab in SwOS, reboot the CRS305, when it comes back up there usually is 'no link' displayed, then tick it again, and either the link comes on then, or another reboot will bring it up. Not exactly intuitive, but I guess the FTLX1475D3BCV dual rate module needs to somehow get initialised or set correctly by the switch first.

Thanks Bertel. I tried these steps and had the same results after each step but could never get a working link established. Because of dbastin's post about advertising 10G I went into RouterOS to see what was going on with that since when I posted about mine being deselected, that was with the 1318 SFP modules working. It was still unselected with these 1475 SFP+ modules but no link.

 

I tried lots of random interface settings while in RouterOS, mostly combinations of flow control on/off/auto, full duplex on/off, auto negotiation on/off but again no go. I booted back into SwOS where auto negotiation was deselected and no go. I enabled auto negotiation, applied new settings (without reboot) and immediately had working 1G links. I didn't have to reboot anything downstream (ER, OM Deluxe, sonic transport or streamer/dac), the links just worked. There was definitely no specific repeatable sequence to get it working but somehow, as you said, the modules finally got initialized. So for now I again have 1G links working between dual rate FTLX1475D3BCV SFP+ modules in the Mikrotik CRS305 (using SwOS) and the regular FTLX1475D3BCL SFP+ modules in my ER and OM. Hopefully it lasts longer than two days this time.

 

If these Mikrotik switches are working "correctly" I'd sure hate to see them working incorrectly.

[89reksal]

1 hour ago, jabbr said:

Honestly I'm confused about what you are trying to do. If the 1318 module is working in the Mikrotik, what is the problem? Why did you get 4 new modules?

I'm beginning to see why we seem to just keep going around in circles with this discussion. It appears you haven't been reading my posts.

 

The 1318 is an SFP module working at 1G in the Mikrotik CRS305 10G switch (and prior to that a Mikrotik CSS610 switch). That configuration has been working all along as stated in multiple posts. I've been asking over the last few pages here how to get an SFP+ module to work at 1G in these switches.

 

I got 4 new modules (4 because I'm connecting 2 devices; ER and OM Deluxe) because you've repeatedly stated that the only way to achieve this is with dual rate 1G\10G modules. So following your advice I decided to get 2 new dual rate 1475 modules for the 2 ports at switch end and to be safe, since you and others had commented about using same modules on each end of a link, I also bought 2 new non dual rate 1475 SFP+ modules for the ER and OM connections.

 

Understand now?

[89reksal]

2 minutes ago, The Computer Audiophile said:

Please excuse me if I missed something of critical importance, but this sounds a bit entitled.

WOW