Optical Network Configurations

[jabbr]

This article is great: https://blog.mellanox.com/2017/03/100g-psm4-the-most-configurable-lowest-cost-single-mode-transceiver-available/

 

I am picking up optical cables including QSFP28 transceivers at each end for <$100 ... again gulp ... so I've been curious in what this technology is... PSM4 (parallel single mode x4) which is a laser split into 4 paths, and then modulated on the silicon chip into 4 x 25 Gbs signals. These cables have 8 single mode fibers (4 duplex).

 

In contrast, the 100GBase-lr4 modules that I am using from my basement to my office multiplex the lasers into 4 different frequencies and then send all 4 down a single single mode fiber! ... and trust me, I'm not spending $8k / module  

 

These integrated silicon photonics chips are bringing the cost of fiberoptic networking down: with careful "shopping" I can afford to do this in my house today, but this technology is being mass deployed and we will all be seeing this (QSFP28 and QSFP-DD connections) in the future. 

 

For "extreme" users this means that NVME storage over the network will be as fast as on a local machine ... we can think of the network as an extension of the PCIe bus.

[The Computer Audiophile]

10 minutes ago, jabbr said:

This article is great: https://blog.mellanox.com/2017/03/100g-psm4-the-most-configurable-lowest-cost-single-mode-transceiver-available/

 

I am picking up optical cables including QSFP28 transceivers at each end for <$100 ... again gulp ... so I've been curious in what this technology is... PSM4 (parallel single mode x4) which is a laser split into 4 paths, and then modulated on the silicon chip into 4 x 25 Gbs signals. These cables have 8 single mode fibers (4 duplex).

 

In contrast, the 100GBase-lr4 modules that I am using from my basement to my office multiplex the lasers into 4 different frequencies and then send all 4 down a single single mode fiber! ... and trust me, I'm not spending $8k / module  

 

These integrated silicon photonics chips are bringing the cost of fiberoptic networking down: with careful "shopping" I can afford to do this in my house today, but this technology is being mass deployed and we will all be seeing this (QSFP28 and QSFP-DD connections) in the future. 

 

For "extreme" users this means that NVME storage over the network will be as fast as on a local machine ... we can think of the network as an extension of the PCIe bus.

Love this!

[jabbr]

[jabbr]

Laying cable is work! QSFP28 with Source Photonics 100Gbase-LR4 end to end from Mellanox SN2700 running Sonic to Dell Precision 5820 W-2245 Workstation

[The Computer Audiophile]

Love it @jabbr

[John769]

anyone aware whether Planet MGB TLX SFP's work in Cisco switches? Probably not but just checking.

[plissken]

Using these on my Cisco core deployments:

 

https://www.fs.com/products/48354.html

 

$99

 

If you are close enough (5 meter and under) you can go with QSFP28 DAC cables and those start at a whopping $39 🙂

[plissken]

On 4/16/2020 at 11:37 AM, jabbr said:

Laying cable is work! QSFP28 with Source Photonics 100Gbase-LR4 end to end from Mellanox SN2700 running Sonic to Dell Precision 5820 W-2245 Workstation

 

Is there a reason your running LR optics in a home vs the cost savings of SR?

[jabbr]

1 hour ago, plissken said:

 

Is there a reason your running LR optics in a home vs the cost savings of SR?

 

Let's see ... 0S2 100m 3.0mm from fs.com is $30 including shipping from China...

 

I picked up 100Gbase-LR4 modules for $100/each ...

 

Alternatively 100Gbase-PSM4 or 100Gbase-SR4 modules for $50 and run MSO  cables ...

 

So no real cost savings anymore ... particularly PSM4 vs SR4 and this way I can run 1/10/25/40/100Gbe over same cable ...

 

I don't need 100m, 50 would probably do, let's say 60m to be safe ... but this way the LR can blast through 2 end/end connectors at the keystone and network end. idk really ... just because

[plissken]

2 minutes ago, jabbr said:

 

Let's see ... 0S2 100m 3.0mm from fs.com is $30 including shipping from China...

 

I picked up 100Gbase-LR4 modules for $100/each ...

 

Alternatively 100Gbase-PSM4 or 100Gbase-SR4 modules for $50 and run MSO  cables ...

 

So no real cost savings anymore ... particularly PSM4 vs SR4 and this way I can run 1/10/25/40/100Gbe over same cable ...

 

I've been seeing new QSFP28 LR for ~$400 and new SR for $100. There isn't a price premium in cabling like there used to be.

 

Are you purchasing used transceivers?

[jabbr]

5 minutes ago, plissken said:

 

I've been seeing new QSFP28 LR for ~$400 and new SR for $100. There isn't a price premium in cabling like there used to be.

 

Are you purchasing used transceivers?

 

Yeah ebay ... you can even get "new" PSM4 or SR4 modules easily at $50 ... I just jumped on the LR4s because of the price but there are many ways that work -- as well as those awesome fiberoptic cables with QSFP28's at each end (Mellanox and others). I got a few for $100 each! I mean ha ha there is zero need to do this but I've been stir crazy ... 

[plissken]

6 minutes ago, jabbr said:

 

Yeah ebay ... you can even get "new" PSM4 or SR4 modules easily at $50 ... I just jumped on the LR4s because of the price but there are many ways that work -- as well as those awesome fiberoptic cables with QSFP28's at each end (Mellanox and others). I got a few for $100 each! I mean ha ha there is zero need to do this but I've been stir crazy ... 

 

I'm halfway through my ACEP (Aruba Certified Edge Professional). It's HPE's equivalent of my CCNP Route/Switch. Bonus is that my employer is paying out bounties because they need this for partner status. It's 6 tests though. Bit of a slog but most of the material is review to my CCNP.

 

I'm good with 10G at home. I get to do plenty of 10/25/40/80/100 at work. I don't even know what I'm going to do with 10G but for $210 in pieces parts why not 😉

[jabbr]

Getting the firmware updated on my 'old' SN2700 (circa 2016) required black magic and then loading SONiC has been another experience, but thank god for open source! I can't even imagine what I'd do with the layer 3 capabilities -- and no idea how to use them! Your field is advancing at an astonishing rate!

[plissken]

5 minutes ago, jabbr said:

Getting the firmware updated on my 'old' SN2700 (circa 2016) required black magic and then loading SONiC has been another experience, but thank god for open source! I can't even imagine what I'd do with the layer 3 capabilities -- and no idea how to use them! Your field is advancing at an astonishing rate!

 

It's crazy. My goal is to turn myself into a NAC and Firewall unicorn. So it's going to be Cisco ISE, HPE Aruba Clear Pass and Fortigate. Couple that with deep dive into Python and that's where I'm going to specialize.

 

We are at the point that there can't be any generalist any longer like 20 years ago.

[John769]

On 4/17/2020 at 6:25 PM, jabbr said:

as well as those awesome fiberoptic cables with QSFP28's at each end (Mellanox and others). 

 

Hi @jabbr is this something worth considering, along with the Cisco 100G SFP's on fs.com? (thanks, @plissken ). The part of my system that has fibre is the Cisco 2960CG -> OM (at endpoint), using single mode Cisco SFP's and standard LC to LC OS2 cables.

 

I'm still interested in trying out a MikroTik CRS305-1G-4S+IN and seeing how it compares to the 2960CG (in switch mode) or OM (in FMC mode) but it's still unclear, from the few mentions in this thread, whether it would be an upgrade. I need to order one and try it out but I was hoping for more feedback by now.  Any general suggestions welcome 😄

 

 

[plissken]

51 minutes ago, John769 said:

 

Hi @jabbr is this something worth considering, along with the Cisco 100G SFP's on fs.com? (thanks, @plissken ). The part of my system that has fibre is the Cisco 2960CG -> OM (at endpoint), using single mode Cisco SFP's and standard LC to LC OS2 cables.

 

I'm still interested in trying out a MikroTik CRS305-1G-4S+IN and seeing how it compares to the 2960CG (in switch mode) or OM (in FMC mode) but it's still unclear, from the few mentions in this thread, whether it would be an upgrade. I need to order one and try it out but I was hoping for more feedback by now.  Any general suggestions welcome 😄

 

 

 

If you want a real upgrade you can get the Cisco 2360's for $50-60 on ebay and they support SFP+ (10GBe). The only caveat is if this is going to be in the same listening room. I fished fiber down to my basement.

 

[jabbr]

No you can't use QSFP28 gear with your system. Short explanation:

 

SFP ports/modules are intended for 1Gbe

SFP(+) ports/modules are intended for 10Gbe but are backward compatible with 1Gbe, that means switches with SFP(+) ports can support SFP modules, there are also dual 10g/1g SFP(+) modules

QSFP ports/modules are intended for 4Gbe, they have 4 parallel 1g lanes per connector -- these hardly exist

QSFP(+) ports/modules for 40Gbe have 4 parallel 10g lanes per connector

 

SFP28 ports/modules support 25 Gbe Ethernet -- they are typically backward compatible with 10 Gbe ... and even 5, 2.5 and 1 Gbe

QSFP28 ports/modules contain 4 parallel 25 Gbe lanes intended for 100 Gbe

QSFP56 for 200 Gbe ... and ...

QSFP-DD with 8 lanes for 200, 400, 800 Gbe

 

My recommendation: if you have the OM as endpoint, you are unlikely to hear a difference with a well made fiberoptic switch. Be happy with your Cisco! The point of fiberoptic isolation is that it is complete electrical isolation. The good switches (10G and up) all have femtosecond grade clocks. The photons are happy  

 

[John769]

@jabbrThanks for summary, very useful!

[John769]

On 4/20/2020 at 2:45 PM, plissken said:

If you want a real upgrade you can get the Cisco 2360'

Hi. Out of interest, what advantage would it have over a microtik crs305?  Cost wise, here in the EU, a s/h cisco might be about the same (ish) as new Microtik.

 

Any crs305 users care to offer more feedback on it versus other fmc's or switches?  One minor observation from the reviews is that it appears to run quite hot, which could be disinsentive in summer if you live in a hot place.  

[plissken]

1 hour ago, John769 said:

Hi. Out of interest, what advantage would it have over a microtik crs305?  Cost wise, here in the EU, a s/h cisco might be about the same (ish) as new Microtik.

 

Any crs305 users care to offer more feedback on it versus other fmc's or switches?  One minor observation from the reviews is that it appears to run quite hot, which could be disinsentive in summer if you live in a hot place.  

 

1: 48 1GBe copper ports

2: 4 SFP+ 10GBe ports. Looks like the crs305 is SFP (1GBe).

3: In the states I can get them for $50-$60 shipped. You'll have to check ebay listings in your neck of the woods.

4. It's Cisco

[ericuco]

59 minutes ago, plissken said:

 

2: 4 SFP+ 10GBe ports. Looks like the crs305 is SFP (1GBe).

 

 

Mikrotik Crs305‑1g‑4s+ has four SFP+ (10GBe) ports.

 

As for the previous comment about it running hot, mine is usually warm to the touch with 4 SPF's connected and active.

[jabbr]

7 hours ago, John769 said:

Any crs305 users care to offer more feedback on it versus other fmc's or switches?  One minor observation from the reviews is that it appears to run quite hot, which could be disinsentive in summer if you live in a hot place.  

Its fanless which is important when located in your audio room — many of the pro grade switches are rather loud 

[John769]

53 minutes ago, jabbr said:

Its fanless which is important when located in your audio room — many of the pro grade switches are rather loud 

 

yes, see what you mean and a worthwhile compromise..  I think the Microtik's looking the likelier candidate if I go SFP+. Would you still recommend the same sfp's you posted when you bought yours a while ago?

 

  

5 hours ago, ericuco said:

As for the previous comment about it running hot, mine is usually warm to the touch with 4 SPF's connected and active.

Thanks.  Are you still happy with yours and do you use it in switch or fmc mode?

[plissken]

5 hours ago, ericuco said:

 

Mikrotik Crs305‑1g‑4s+ has four SFP+ (10GBe) ports.

 

As for the previous comment about it running hot, mine is usually warm to the touch with 4 SPF's connected and active.

 

Gotcha, I just looked up the model that was posted (not the +). For $124 U.S. It's a no brainer. FS.COM 10GBe SR LC modules should be $20.

[jabbr]

40 minutes ago, John769 said:

Are you still happy with yours and do you use it in switch or fmc mode?

it’s hard to go wrong with a 10Gbe switch. I don’t use FMCs because all my endpoints are either SFP or wireless in.