Cost effective 10G home network upgrade

[plissken]

I setup 4088kb Jumbo Frame and as suspected it didn't do anything for me CPU wise since the NIC's offload all the processing.

 

Plus getting a frame size that different manufacturer cards commonly support can be a PITA.

 

 

[jabbr]

On 2/9/2020 at 1:44 PM, plissken said:

I setup 4088kb Jumbo Frame and as suspected it didn't do anything for me CPU wise since the NIC's offload all the processing.

 

Plus getting a frame size that different manufacturer cards commonly support can be a PITA.

 

 

 

Yes, there's so much bandwidth, and the cards do offload so I keep it really simple.

 

The cards usually also offload network boot/PXE and/or iSCSI, and while my Windows is sketchy, I've iSCSI booted Linux. Its pretty obvious that Linux runs in memory (at least mostly) because you can detach and reattach the cable and the iSCSI booted device keeps humming along.

[plissken]

@jabbr

 

What the highest throughput you've seen and what was the hardware configuration?

[jabbr]

On 2/11/2020 at 11:02 AM, plissken said:

@jabbr

 

What the highest throughput you've seen and what was the hardware configuration?

Not a simple question. As you go above 10Gbe, then PCIe becomes a bottleneck assuming the NIC can write directly to CPU cache (eg DirectIO) ...

 

So PCIe 3.0 x 8 = 64 Gb/s, essentially saturated by 40Gbe card including overhead. 
 

Now you need something to generate that IO, and certain database/infetencing can burst at that rate. 
 

So the Dell with Mellanox ConnectX-4 card memory to memory can burst ... let’s say 20 Gbs?? It’s very bursty so the longer you measure, the lower the average throughput. 
 

I’m looking for PCIe 4.0 and lots of lanes to re up the performance — the GPU eats up x16 ... then the storage array etc.

 

(none of this is relevant to audio folks ;) 

[jabbr]

Let me add that with a ZFS based NAS with lots of RAM, that you are loading the database into RAM, so the network can approach saturation. That said, it’s hard to need to pull >20 Gbs across the network at home 😂

 

I think the use case for 100 Gbe will be arrays of NVME over Ethernet but again you need very serious CPU/GPU along with gobs of IO lanes to deal with this data.

[plissken]

All good, I was just asking what you've managed at home. I understand the various bottlenecks not-withstanding.

[jabbr]

Yeah I’ll say 20-30 Gbs 🤷🏻‍♂️

[plissken]

BTW the NC532 is 17 watts nominal, 19 watts average. There is the NC552 that is About $10-$12 more ($38) that is 11 watts. So appreciably less thermals.

 

 

[jabbr]

23 minutes ago, plissken said:

BTW the NC532 is 17 watts nominal, 19 watts average. There is the NC552 that is About $10-$12 more ($38) that is 11 watts. So appreciably less thermals.

 

 

Yes!!!

 

Fiber typically uses less power than copper Ethernet, and the power usage does not increase like copper with speed. Fiber Ethernet is used in big boy data centers to reduce power consumption and heat. 
 

The Intel 520/710 series NICs use 3-8W  

 

https://www.servethehome.com/qsfp-v-sfp-v-10gbase-t-testing-power-consumption-differences/

[plissken]

13 minutes ago, jabbr said:

Yes!!!

 

Fiber typically uses less power than copper Ethernet, and the power usage does not increase like copper with speed. Fiber Ethernet is used in big boy data centers to reduce power consumption and heat. 

 

When we do SFP/SFP+ blades and we have optical/copper mix we stagger the SFP's just for this reason.

[plissken]

55 minutes ago, jabbr said:

The Intel 520/710 series NICs use 3-8W

 

Easily worth the $60... May have to consider just because.

[plissken]

On 2/13/2020 at 10:50 AM, jabbr said:

The Intel 520/710 series NICs use 3-8W

 

One on it's way... Are these going to want a specific SFP+ module?

[jabbr]

10 minutes ago, plissken said:

 

One on it's way... Are these going to want a specific SFP+ module?

https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

there are also generics but I use the FTLX (also AFBR)

[plissken]

FS has a compatible module also

 

SFP-10GSR-85

[jabbr]

Yes I use FS. I like the fact that these modules are 10G but run also at 1G.

[lpost]

On 2/6/2020 at 6:29 AM, plissken said:

Started setting up a cost effective 10G Fiber SR network.

 

My file server is a Dell Power Edge R620 and upgraded to a Broadcom 2 GBE / 2 SFP+ card for $20. BTW the R620's are quite affordable (often $200 to start) on eBay. Add drives of your choice up to 8 drive bays.

Cisco WS-2930 48 port 10/100/1000 copper and 4 SFP+ $65

HP NC522SFP Dual PCI-E X 8 $25

 

SFP and cabling (all other MM OM3 LC<>LC I already have) $100

 

So for $210 you can get 10G fiber for a single end point and a server.

 

 

 

 

 

Just a heads up, the HP NC522SFP is one really hot running card and is not compatible with AMD processors at all.  Like burn your fingers hot...the Intel X520 DA2 is alternative <$40.

[lpost]

On 2/6/2020 at 3:27 PM, Foggie said:

Based on the title 10GB, isn’t this switch only 1GB capable?  I’ve been contemplating updating my managed switch and internal wkst with10GBe fiber 

 

SFP+ denotes 10Gbps. 

[lpost]

9 hours ago, plissken said:

 

One on it's way... Are these going to want a specific SFP+ module?

 

They do, unless you're running Linux, their is a driver override.  FS.com sells one Intel compatible SFP.  

[jabbr]

18 hours ago, lpost said:

 

They do, unless you're running Linux, their is a driver override.  FS.com sells one Intel compatible SFP.  

You can pass a flag to the driver to allow unsupported sfp. 

[cjf]

Are folks setting up 10gb at home "Just Because" or is there some other requirement that is the motivation?

I'm using basic 1GB connectivity for the internal LAN and see the following speeds between, in this example, my DAW and my NAS pretty much at all times.

 

For a poor slob like myself this seems more than fast enough

 

 

[plissken]

On 3/16/2020 at 10:51 PM, cjf said:

Are folks setting up 10gb at home "Just Because" or is there some other requirement that is the motivation?

I'm using basic 1GB connectivity for the internal LAN and see the following speeds between, in this example, my DAW and my NAS pretty much at all times.

 

For a poor slob like myself this seems more than fast enough

 

 

 

This culminated out of the if $640 audiophile switch did nothing and it capped me at 11MB/s what could I do with the refund?

 

So it only cost me $210 to go with straight up enterprise class hardware and swing from 11MB/s to 332MB/s and I have an isolation 'Moat' that is 15 meters long.

 

Running fiber is just as easy as running copper and totally affordable for treating 'audiophile nervousa' where there is a jitter gremlin hiding in every darkened port.

 

10GBe has extremely tight jitter requirements (for data transmission that is).

[cjf]

Sounds good and makes sense. I guess the audiophile switch was too pre-occupied with removing veils to be bothered with things like passing packets 🤣

[lucretius]

On 3/16/2020 at 10:51 PM, cjf said:

Are folks setting up 10gb at home "Just Because" or is there some other requirement that is the motivation?

 

I haven't set up anything yet.  I have 1.5 Gb internet connection and would like to use it all for my PC.  Unfortunately, If I remove the ISP's modem/router, I will lose "home phone" (I could take out the SFP module and plug it into another device -- nonetheless, the ISP will not give out the passwords for connecting the phone) and this router has only 1 Gb connections on LAN side.

[jabbr]

16 hours ago, lucretius said:

 

I haven't set up anything yet.  I have 1.5 Gb internet connection and would like to use it all for my PC.  Unfortunately, If I remove the ISP's modem/router, I will lose "home phone" (I could take out the SFP module and plug it into another device -- nonetheless, the ISP will not give out the passwords for connecting the phone) and this router has only 1 Gb connections on LAN side.

Seems to me that if your ISP is actually selling you 1.5 Gb then you need a router that delivers that. 

[lucretius]

3 hours ago, jabbr said:

Seems to me that if your ISP is actually selling you 1.5 Gb then you need a router that delivers that. 

 

The router can deliver near 1.5Gb bandwidth accumulated across multiple connections. I got this speed as part of a promotional deal and mainly was interested in the advertised upload speed of 940 Mbps so I can do backups to the cloud. As it turns out, I cannot find a cloud service that allocates/delivers anywhere near this much speed to a single customer.  The 1.5Gb/940 Mbps bandwidth is wasted on me. I'll probably lower it to 500 Mbps/500 Mbps and save a few dollars.

 

I'm sure the ISP has speeds even higher on the roadmap.  By then, I'm hoping they give customers a modem/router that has SFP ports on the LAN side.