Cost effective 10G home network upgrade

[jabbr]

Yes the Dell 620 is a terrific option (I got once / similar based on your advice) cheap enough that I’ve used it as a prototype/testbed server. I upgraded to a Mellanox 100Gbe QSFP28 NIC which is overkill/pointless for audio! For the ultimate in HQPlayer upsampling then clock speed is important. 
 

10Gbe is indeed dirt cheap and great quality and you’ve redemonstrated that it’s pretty much plug and play with diverse types of equipment. 

[jabbr]

On 2/9/2020 at 1:44 PM, plissken said:

I setup 4088kb Jumbo Frame and as suspected it didn't do anything for me CPU wise since the NIC's offload all the processing.

 

Plus getting a frame size that different manufacturer cards commonly support can be a PITA.

 

 

 

Yes, there's so much bandwidth, and the cards do offload so I keep it really simple.

 

The cards usually also offload network boot/PXE and/or iSCSI, and while my Windows is sketchy, I've iSCSI booted Linux. Its pretty obvious that Linux runs in memory (at least mostly) because you can detach and reattach the cable and the iSCSI booted device keeps humming along.

[jabbr]

On 2/11/2020 at 11:02 AM, plissken said:

@jabbr

 

What the highest throughput you've seen and what was the hardware configuration?

Not a simple question. As you go above 10Gbe, then PCIe becomes a bottleneck assuming the NIC can write directly to CPU cache (eg DirectIO) ...

 

So PCIe 3.0 x 8 = 64 Gb/s, essentially saturated by 40Gbe card including overhead. 
 

Now you need something to generate that IO, and certain database/infetencing can burst at that rate. 
 

So the Dell with Mellanox ConnectX-4 card memory to memory can burst ... let’s say 20 Gbs?? It’s very bursty so the longer you measure, the lower the average throughput. 
 

I’m looking for PCIe 4.0 and lots of lanes to re up the performance — the GPU eats up x16 ... then the storage array etc.

 

(none of this is relevant to audio folks ;) 

[jabbr]

Let me add that with a ZFS based NAS with lots of RAM, that you are loading the database into RAM, so the network can approach saturation. That said, it’s hard to need to pull >20 Gbs across the network at home 😂

 

I think the use case for 100 Gbe will be arrays of NVME over Ethernet but again you need very serious CPU/GPU along with gobs of IO lanes to deal with this data.

[jabbr]

Yeah I’ll say 20-30 Gbs 🤷🏻‍♂️

[jabbr]

23 minutes ago, plissken said:

BTW the NC532 is 17 watts nominal, 19 watts average. There is the NC552 that is About $10-$12 more ($38) that is 11 watts. So appreciably less thermals.

 

 

Yes!!!

 

Fiber typically uses less power than copper Ethernet, and the power usage does not increase like copper with speed. Fiber Ethernet is used in big boy data centers to reduce power consumption and heat. 
 

The Intel 520/710 series NICs use 3-8W  

 

https://www.servethehome.com/qsfp-v-sfp-v-10gbase-t-testing-power-consumption-differences/

[jabbr]

10 minutes ago, plissken said:

 

One on it's way... Are these going to want a specific SFP+ module?

https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

there are also generics but I use the FTLX (also AFBR)

[jabbr]

Yes I use FS. I like the fact that these modules are 10G but run also at 1G.

[jabbr]

18 hours ago, lpost said:

 

They do, unless you're running Linux, their is a driver override.  FS.com sells one Intel compatible SFP.  

You can pass a flag to the driver to allow unsupported sfp. 

[jabbr]

16 hours ago, lucretius said:

 

I haven't set up anything yet.  I have 1.5 Gb internet connection and would like to use it all for my PC.  Unfortunately, If I remove the ISP's modem/router, I will lose "home phone" (I could take out the SFP module and plug it into another device -- nonetheless, the ISP will not give out the passwords for connecting the phone) and this router has only 1 Gb connections on LAN side.

Seems to me that if your ISP is actually selling you 1.5 Gb then you need a router that delivers that.