Optical Network Configurations

[pl_svn]

whilst at it... anyone (@Superdad?) knows if this will work with the EtheRegen? 😶

 

thanks

[jabbr]

6 minutes ago, pl_svn said:

whilst at it... anyone (@Superdad?) knows if this will work with the EtheRegen? 😶

 

thanks

 

Those are sweet cables and I use the 100Gbe versions with my Mellanox switch. Likely worth a try. I haven't specifically tested that brand cable with the EtheRegen

[jabbr]

32 minutes ago, R1200CL said:

I feel a bit responsible for starting this latest fiber optics discussion. 
I think the subject has been enlightening enough now to not making any more confusion. 
 

I did based on knowledge from this tread as well as internet search, found several sources stating it won’t work. Hence I was afraid @AfterDark. started something that could turn out to be problematic. 

 

The reports of people saying that it won't work are why we need actual reports of what actually works with what...

 

32 minutes ago, R1200CL said:

Now we know better. And the discussion would happen sooner or later. I think better up front. Now we just have to wait for that one guy that won’t be able to make this 10GB FTLX1475D3BTL to work in his specific setup 😃

Edit:

@jabbr SFPQ28 / SFP28 fits SFP cages ?

 

They might work, might be even likely to work, but I haven't tried.

[R1200CL]

29 minutes ago, pl_svn said:

whilst at it... anyone (@Superdad?) knows if this will work with the EtheRegen? 😶

 

thanks

To answer this correctly, one must know what’s in the other end. With a Mikrotik switch it won’t as it’s a 10GB SFP+. With a Cisco 2960 or almost any FMC, it should work. 
This the area I myself misunderstood.  Hopefully I’ve got it right now. 
 

Orange cable indicate multi mode fiber. Type of laser unknown. 
I assume the modules itself are dual rate. As we now know, that doesn’t matter. 
 

FTLF1421P1BCL (or FTLF1421P1BTL) with yellow cable will always work. They are cheap on eBay. 
 

[plissken]

35 minutes ago, pl_svn said:

whilst at it... anyone (@Superdad?) knows if this will work with the EtheRegen? 😶

 

thanks

 

Looking at that part you might as well try it for $49. If you go the SX route with OM2 cabling your at less than 1/2 the cost though with FS.com SX transceivers being ~$6.

[The Computer Audiophile]

1 hour ago, jabbr said:

My objection was with the statement "ALL SFP+ modules will work in SFP cages" 

Is it really necessary to get all whipped up over that? Nobody would suggest ALL and mean ALL. The there’s a product somewhere that won’t work. Nobody reads that and really thinks ALL means ALL ALL the time. 
 

If we need to caveat ALL our speech at ALL times, count me out. 

[R1200CL]

42 minutes ago, jabbr said:

No QSFP28 modules most certainly do not fit SFP ports.

Ops. 
I think I just may created some confusion.

https://www.optcore.net/archive25874/
 

SFP28 is compatible with SFP’s.

 

I think you’re saying if one do happen have a QSFP28 in one end, like you do, you could have SFP/SFP+/SFP28 in the other end. Which even could be the EtherRegen. 

[jabbr]

12 minutes ago, The Computer Audiophile said:

Is it really necessary to get all whipped up over that? Nobody would suggest ALL and mean ALL. The there’s a product somewhere that won’t work. Nobody reads that and really thinks ALL means ALL ALL the time. 
 

If we need to caveat ALL our speech at ALL times, count me out. 

To be clear the specification does not intend than SFP+ modules be put in SFP cages. Some SFP+ modules draw too much power and I envision the possibility that putting the wrong SFP+ module in an SFP cage could overheat or even damage the device. 
 

You can scour the internet but it’s not generally recommended. I don’t mind breaking the rules, and yes I tested doing this and reported my results. 
 

Im hardly whipped up but because of the source, I wanted to be crystal clear in my statements. I am very much an advocate of testing specific modules in specific configuration. I do feel it’s necessary despite statements otherwise.

[ted_b]

9 minutes ago, R1200CL said:

Ops. 
I think I just may created some confusion.

Lately, the level of confusion and misspeak/typos in this thread (and somewhat in the latest posts on EtherRegen) is overwhelming.  We newbies look to these threads for definitive direction.  Let's please make sure we know about what we say (unless it's a question, of course) before posting it. 

 

I got into fiber/optical several years ago before my move across country, and have a box full of converters, fiber and some SFPs (no idea if they are + or not).  Without some empirical guidance people like me are forced to ask very specific questions, cuz the latest conventional wisdom doesn't seem to be rising to the top. 

 

Confused in the Rockies  :)

Ted

[pl_svn]

32 minutes ago, R1200CL said:

With a Mikrotik switch it won’t as it’s a 10GB SFP+

yup, was thinking about getting one (or two) of those Mikrotik CSS610-8G-2S+IN switches
I'll stay with my TP-Link T2500G-10TS and single mode fiber to the ER, then

 

thank you

[jabbr]

1 hour ago, ted_b said:

Lately, the level of confusion and misspeak/typos in this thread (and somewhat in the latest posts on EtherRegen) is overwhelming.  We newbies look to these threads for definitive direction.  Let's please make sure we know about what we say (unless it's a question, of course) before posting it. 

 

I got into fiber/optical several years ago before my move across country, and have a box full of converters, fiber and some SFPs (no idea if they are + or not).  Without some empirical guidance people like me are forced to ask very specific questions, cuz the latest conventional wisdom doesn't seem to be rising to the top. 

 

Confused in the Rockies  :)

Ted

 

Ted, how can we help! First of all, the modules slightly change their wavelength's based on elevation so depending on how high up you are in the Rockies you need to adjust your equipment ... there is a tiny little screw that you are going to need good loupes to see ... 😝 just kidding...

 

Yeah so let me update you on my thinking from the early days in a nutshell... some of those FMCs and some of the really cheap SFP switches are well, not the best designs. They work fine for the job they were designed for, mostly, but I am not going to say that the signal integrity is perfect... so there are a few  things:

 

1) the 10Gbe equipment is uniformly designed with vastly better attention to signal integrity e.g. board layout and the 10Gbe specifications **require*** end to end signal integrity measurements. SFP+ modules use better lasers, some of the single mode SFP+ modules use the best lasers. SFP28 modules (25Ghz) use even better lasers. So way back when I recommended the Finisar SFP+ modules that I tested in SFP ports ... those recommendations stand. Way back I was using Intel x520 NICs in low powered Celeron endpoints. Thats still a good recommendation. I had played with the old Mellanox NICs but some overheated for me. I am currently using the newer Mellanox NICs but these need a fan for cooling (or a good fanless cooling system).

2) Now there are audiophile designed endpoints  with an SFP port from several companies including Lumin and Sonore. This is terrific. I still use my Clearfog Base as an NAA. For whatever reason it hasn't become popular, probably because you need to get it up and running with Linux. There is also another device that I haven't tested.

3) The EtherRegen also has an SFP port. I fully expect that even though it is a 1Gbe device that if it were tested according with the 10Gbe tests, e.g. the stressed receiver test, that it would similarly prevent phase noise from propagating across a network, in a similar fashion to the 10Gbe equipment. The EtherRegen is also designed to limit common mode noise transmission across a copper ethernet link -- that's not an issue with fiberoptic endpoints but most endpoints are still copper.

4) I have transitioned to single mode in my house because 100Gbe can be transmitted on a duplex LC-LC single mode cable, whereas multimode cabling is entirely different.

5) FMCs ... my old FMCs are sitting in a box. I don't recommend the cheap FMCs anymore, rather I would either get a Sonore opticalModule or cheap switch with SFP+ ports such as one of the Mikrotiks.

6) I can't detect an SQ improvement in different 10Gbe NICs nor with anything above 10Gbe, so there is, according to my ear, a limit beyond which ethernet based phase error is audible -- if its audible at all. My own view is that any improvement in fiberoptic is due to the elimination of common mode noise. Other people are free to try to hear a difference for themselves. Along with this, there is ZERO need to try to alter the clock in a good network switch. I have no doubt that the Israeli engineers at Mellanox who have graciously helped me upgrade my firmware and intall SONiC (an open source and free Linux for network switches!!!) would immediately block me and disown the fact that they have every spoken with me if I called with a question about how to fix my switch after altering its clock!! Dont even try this!

 

Anything else?

 

7) those fiberoptic cables with an SFP+ connector at each end are **sweet** (mine have a QSFP28 connector at each end and I picked them up for really cheap).

[R1200CL]

@ted_b

You can hardly be defined as a newbie 😀

And I do think the latest wisdom is finally quite clear to most of us. And it’s important to have this discussion at an early stage. 

If any of John’s devices may get a shorter life or a get slightly hotter  due to increased power draw, I’m sure we would be warned by now.

@AfterDark.  as a huge Uptone dealer has gotten @Superdad as well as @JohnSwenson blessings to sell and promote that specific FTLX1475D3BTL together with EtherRegen. 
So everything should be good. 

[jabbr]

2 minutes ago, R1200CL said:

@ted_b

You can hardly be defined as a newbie 😀

And I do think the latest wisdom is finally quite clear to most of us. And it’s important to have this discussion at an early stage. 

If any of John’s devices may get a shorter life or a get slightly hotter  due to increased power draw, I’m sure we would be warned by now.

@AfterDark.  as a huge Uptone dealer has gotten @Superdad as well as @JohnSwenson blessings to sell and promote that specific FTLX1475D3BTL together with EtherRegen. 
So everything should be good. 

 

That's terrific ... the specific part listed is the single mode equivalent of the multimode part I listed on the very first post of this thread, and those have been running ever since -- not too hot!!. and reminds me:

 

8) people ask about x,y or z brand that they got and I've dug deep on the spec sheets, looking at what lasers, power draw etc for Finisar which publishes this so in the same way I've been "burned" with "generic" switches and counterfeit/generic NICs, I just can't recommend brands that I don't know.

[Superdad]

41 minutes ago, jabbr said:

Yeah so let me update you on my thinking from the early days in a nutshell...

Really terrific and positive post!

Like that spirit and tone. 

Thanks and have a great weekend,

-Alex C.

[charlesphoto]

59 minutes ago, jabbr said:

 

 

 

7) those fiberoptic cables with an SFP+ connector at each end are **sweet** (mine have a QSFP28 connector at each end and I picked them up for really cheap).

Until you have to try and get them through an inaccessible hole in the floor behind a wall that also has other cables running through it that is! 😅

[jabbr]

6 minutes ago, charlesphoto said:

Until you have to try and get them through an inaccessible hole in the floor behind a wall that also has other cables running through it that is! 😅

yeah they are relatively short and I wouldn't use for in-wall installation. The advantage of the regular single mode cable is that the same cable is used from 1g -> 100g and up

[ted_b]

Thanks so much for the summary core dump!  I will list what I have once I head downstairs and find the moving box labeled "fiber".  😎  I will assume most of it is throw away, but maybe that switch I had is still good. 

 

This next point is really a post I need to put in Alex's thread:  I am going to be doing 1.5Mhz PCM and DSD1024 (maybe), but definitely DSD256 multichannel, so I am weary of setting up an EtherRegen in its normal a->b setup.  100mbps won't cut it.  But that's not to say the EtherRegen is out.  Maybe it's the best thing I can use, going b->a?  Dunno yet. 

 

OK, I'm not a newbie, but anything I learned back in 2015 has atrophied or vanished from non-use.  😎

[R1200CL]

About jitter requirements. 
https://grouper.ieee.org/groups/802/3/ae/public/may01/ewen_1_0501.pdf

It’s an old paper, but this was the requirement I was able to find. 

 

Just google “jitter 802.3ae requirements” if you like to dive into this. 
 

Is also seems specification varies a bit depending on use.

 

I expect the RIN (Relative Intensity Noise) number is important to. This is where FTLX1475D3BTL has better numbers. RIN 1420 is 120. 

 

 

With so close numbers in the data sheet between FTLX1475D3BTL and FTLF1421P1BCL one can assume you won’t hear any difference, like jabbr indicates. 
 

[Duke40]

10 hours ago, jabbr said:

7) those fiberoptic cables with an SFP+ connector at each end are **sweet** (mine have a QSFP28 connector at each end and I picked them up for really cheap).

 

Thanks 🙏 for your posts, it helps many people 👍 (well, it certainly has helped me! 😀).

 

This cable does not seem to have been tried yet but I am feeling lucky and a little adventurous, so I just purchased 2 of these SFP+ Active Optical Cables made by MikroTik:

"Mikrotik S+AO0005 SFP+ 5m 10-Gigabit Fiber Channel Direct Attach Cable"

https://mikrotik.com/product/s_ao0005

To connect from MikroTik CRS305 switch to each of my EtherREGENs.

These MikroTik fiberoptic cables with SFP+ connector at each end are very reasonably priced, only about $US45 each, I am happy with the MikroTik switch so decided to stay with the same brand for these Active Optical Cables.

 

 Have setup my MikroTik CRS305 switch for "Auto Negotiation" on all ports, if I have any trouble I'll just force them to 1G in MikroTik SwitchOS.

[fds]

On 3/18/2021 at 5:31 PM, fds said:

A pair of FTLX1475D3BCV transceivers has just arrived. Looking forward to try them with the CSS610 MikroTik switches later. This will be my first use of a 10GB fiber connection in my audio system. Will report back.

Last night I had my first listening session using the FTLX1475D3BCV transceivers inside the CSS610 MikroTik switches. While so far I did not do any A/B comparisons, my first impressions were very, very positive. Somehow I had the feeling that bass punch, timing and structure was improved, maybe timing in general, along with blacker backgrounds. In fact, I am quite happy with what I had experienced (despite simply using the cheap SMPS units to power the two CSS610). Will continue to test this along with other configurations (e.g. using one CSS610 only directly connected to the oR instead of the oM in between, better power supply units/power banks on the CSS610 and thereby avoiding to pollute the mains, replace Planet Tech SFPs with Finisar SFPs in oM/oR, etc.).

 

Here my used setup: Normal PS/Mac -> Supra CAT8 -> CSS610 with FTLX1475D3BCV -> Single Mode Fiber running at 10G -> CSS610 with FTLX1475D3BCV -> Sablon 2020 CAT cable -> oM with Planet Tech SFP transceiver -> Single Mode Fiber running at 1G -> oR with Planet Tech SFP transceiver -> Sablon 2020 USB cable -> T+A DAC8DSD at fixed volume setting and volume controlled in software -> mono power amps -> ...

 

 

[jabbr]

12 hours ago, R1200CL said:

About jitter requirements. 
https://grouper.ieee.org/groups/802/3/ae/public/may01/ewen_1_0501.pdf

It’s an old paper, but this was the requirement I was able to find. 

 

Just google “jitter 802.3ae requirements” if you like to dive into this. 
 

Is also seems specification varies a bit depending on use.

...

 

I expect the RIN (Relative Intensity Noise) number is important to. This is where FTLX1475D3BTL has better numbers. RIN 1420

With so close numbers in the data sheet between FTLX1475D3BTL and FTLF1421P1BCL one can assume you won’t hear any difference, like jabbr indicates. 
 

 

The table discussed various applications for the Si 5334 clock generator device ... apropos the value of an external 10 Mhz clock reference, from the datasheet:

[1] For optimal jitter and phase noise performance, Silicon Labs requires the use of 48 to 54 MHz XTALs
compliant with the specifications noted in the Si534x/8x Family Reference Manuals. Using XTAL frequencies
other than these may be possible, but jitter and phase noise performance may be degraded. Contact Silicon
Labs to determine the impacts on your configuration.

So, the idea of randomly plopping in an external clock to "improve" a high quality network switch is ludicrous.

 

You can see that the ***end to end*** jitter measurements are in the 0.28 UI or 2ps range and jitter comes from the device electronics not just the clock. That means the device is required to reject external jitter. The network does not allow jitter to propogate from one device to another. I *do nothing* to jitter optimize a server except use a good NIC which is required to also meet these specifications regardless of how crappy an ATX supply is being used for it. Now if you don't consider 10Gbe to be tight enough, consider 25 Gbe where the UI is 1/2.5 the size and consequently the end to end jitter budget is in the femtosecond range. That's insane. Thats with SMPS supplies. That's worst case, being measured with injected jitter. There is magic going on in them there networks. 

 

In any case for the folks at home, you don't need to understand all the nitty gritty details of why this works, it just works. Noise does not make it across a modern fiberoptic network.

[R1200CL]

10 hours ago, Duke40 said:

if I have any trouble I'll just force them to 1G in MikroTik SwitchOS.

That’s probably a nice way to still use 10GB modules in both ends. Thanks for sharing. I will also test when my Mikrotik arrives. 

[R1200CL]

On 3/20/2021 at 4:25 PM, Superdad said:

In the context of using SPF+ transceivers in a 1Gbe device, you can’t have one device with an 10Gbe SFP+ cage and the other be just SFP.

I definitely shall test this 😀

Using the $99 Mikrotik switch. 
 

If not @Duke40 or @fds will post results before me. 
 

However I do agree that this is the general rule, and the reason is that a 10GB switch getting a 10GB module, will of cause transmit 10GB to a 1GB device. Won’t work. 

 

The more interesting question is if it’s sufficient enough (SQ wise) to just use the anticipated best 10GB module in EtherRegen or opticalRendu, and use a reasonable priced 1421 in the 10GB switch. I think this approach is supported by @jabbr 

On 3/20/2021 at 5:15 PM, jabbr said:

I think the only thing that really matters is the endpoint. As long as the switch is compliant to the very strict end-to-end phase noise measurements, then the quality of the endpoint SFP might matter as its electrically connected to the endpoint. 

 

I guess all this could be measured using eye pattern testing. It may only be of academic interest. 
As my understanding is that EtherRegen necessary do not comply with some sort of a 10GB standard, I assume such a test is meaningless. But I have no idea. 
I think also it’s not an easy task to perform. 
There is a limit to when we should stop chasing best fiber optics in order to achieve better SQ.

As there is probably a limit to how good phase noise measurements you really need. 

 

 

[jabbr]

31 minutes ago, R1200CL said:

 

However I do agree that this is the general rule, and the reason is that a 10GB switch getting a 10GB module, will of cause transmit 10GB to a 1GB device. Won’t

 

Ethernet switch ASICs do SERDES which means they accept packets over a 10Gbe link, buffer and then retransmit at 1Gbe if needed. Honestly if you’d stop speculating and simply connect things you will see that you can connect a server via a 10,40 or 100Gbe link into a switch and connect to a 1Gbe link to and endpoint and it works, as I’ve actually do.
 

31 minutes ago, R1200CL said:

The more interesting question is if it’s sufficient enough (SQ wise) to just use the anticipated best 10GB module in EtherRegen or opticalRendu, and use a reasonable priced 1421 in the 10GB switch. I think this approach is supported by @jabbr 

 

I guess all this could be measured using eye pattern testing. It may only be of academic interest. 
As my understanding is that EtherRegen necessary do not comply with some sort of a 10GB standard, I assume such a test is meaningless. But I have no idea. 

 

Is your understanding based on something published by @Superdad? Or are you speculating? Obviously the ER is a 1Gbe device so doesn’t claim to be 10Gbe. I have no knowledge that they have done the stressed receiver test.

31 minutes ago, R1200CL said:

I think also it’s not an easy task to perform. 

There are labs that have the testing equipment but I don’t know if the automated equipment which does these tests eg Tektronix or Keysight, is programmed to test at 1Gbe speeds.

31 minutes ago, R1200CL said:

There is a limit to when we should stop chasing best fiber optics in order to achieve better SQ.

As there is probably a limit to how good phase noise measurements you really need. 

The limit to the needs of “best fiberoptics” is precisely the point where every last bit of noise either phase, voltage, differential or common mode, has been scrubbed from the server. A good 1Gbe Ethernet is likely to do it, but pick whatever server noise level you think you can possibly hear, and I can pick a fiberoptic Ethernet link that will scrub that. It’s physics.  

[jabbr]

2 hours ago, R1200CL said:

That’s probably a nice way to still use 10GB modules in both ends. Thanks for sharing. I will also test when my Mikrotik arrives. 

Again and again folks ... use the dual 10g/1g devices on the switch side. 
 

People are getting confused.

In order to use 10Gbe capable modules with 1Gbe links: 

1) use a dual 10/1 Gbe SFP+ on the 10Gbe switch side

2) use the tested 10Gbe modules on the SFP side

(Finisar has 10/1 and known good SFP+ modules) 

3) if using a 1Gbe SFP switch then use the known good SFP+ module

 

In some cases a 10Gbe SFP+ switch will auto negotiate to 1Gbe despite 10g SFP+ modules on both sides but this is not standard behavior so you need to test it.