Building a DIY Music Server with custom made parts

[jabbr]

The newest architecture which bears looking into and which I am running is the Xeon W - LGA1200 / 422. These really are ideal, in terms of Xeons, for audio workstations because of:

 

1) high base clock rates

2) multiple PCIe lanes

3) ECC RAM / quad channel 

 

Now, I am not saying that this is better than the 470 series (non ECC) and ASUS makes both, but for example my 100 Gbe NIC as well as RTX 2080 GPU ... both now NVIDIA and both requiring PCIe x16 ... 

 

Arguably if one wants the very best clocking then wait for PCIe-4.0 motherboards — Intel is still at PCIex3.0 because the current boards can’t handle the right jitter requirements of PCIe 4.0

[jabbr]

27 minutes ago, OAudio said:

 

I didn't claim any experience running the EC modulators I don't know them (I think I mentioned them as DSPs in my post) so defiantly take your views on that.

 

I am a relatively new user of this forum so its understandable to assume that I may by putting forward baseless ideas or opinion about the 621 architecture for instance. I want to reassure you that the comments are not purely speculation and are in fact based on many, many years of research both practical and paper based into a number of Intel Architectures. For sure I would say I am not your typical server builder. Where I haven't had direct experience of something you will see that I have been calling this out in my posts and then may offer opinion or just leave the matter as I cannot add any value.

Hello OAudio, 

 

The 621 is a fine architecture and is well tested and I have one. My prior workstation. It simply wasn’t capable of doing the math necessary to run the best upsampling algorithms. My new 422 based workstation sounds better to me. 
 

That is my experience. There are different ways to do things so I don’t get tied to a specific version of Intel because they are always working on the next best greatest chip

 

But if Peter wants to design a cooling system for one board — I would pick the ASUS 470 (one if them) because the i9-10900k is going to be the chip that most people building new workstations are going to want to use (or ASUS WS 422 if Xeon desired) 

[OAudio]

7 minutes ago, jabbr said:

The newest architecture which bears looking into and which I am running is the Xeon W - LGA1200 / 422. These really are ideal, in terms of Xeons, for audio workstations because of:

 

1) high base clock rates

2) multiple PCIe lanes

3) ECC RAM / quad channel 

 

Now, I am not saying that this is better than the 470 series (non ECC) and ASUS makes both, but for example my 100 Gbe NIC as well as RTX 2080 GPU ... both now NVIDIA and both requiring PCIe x16 ... 

 

Arguably if one wants the very best clocking then wait for PCIe-4.0 motherboards — Intel is still at PCIex3.0 because the current boards can’t handle the right jitter requirements of PCIe 4.0

Ver interesting,

 

I have had my head down working on the c621 and had missed the 422 I will pull the documents and take a look.

 

I think I understand our different level of emphasis on PCIe now with your network and graphics setup.

 

I have gone down the route of all interfaces on-chip on the PCH, so network, USB, and storage.  I think this gives CPU > DMI > PCH > direct on chip to USB / and SATA / and Network (the network uses another serial standard from PCI chip to a media controller on the motherboard, I forget what its called, but I think its still in the PCH network clock domain which is nice).

 

If I have this right I don't think I use PCIe this way at all (must check again ) but it puts me more at the mercy of the motherboard implementation of the relevant ports for electrical signal integrity of the ports. Still the clocking is good for the reasons mentioned earlier.

 

You seem to have quite a collection of systems

 

[Dev]

17 minutes ago, jabbr said:

I would pick the ASUS 470 (one if them) because the i9-10900k is going to be the chip that most people building new workstations are going to want to use (or ASUS WS 422 if Xeon desired) 

 

The Xeon equivalent of 1200 is W480. 422 is still 2066.

[jabbr]

11 minutes ago, Dev said:

 

The Xeon equivalent of 1200 is W480. 422 is still 2066.

I misstated that: the W-2200 series is 2066! The new i9-10900k is 1200.

[StreamFidelity]

35 minutes ago, jabbr said:

The 621 is a fine architecture and is well tested and I have one. My prior workstation. It simply wasn’t capable of doing the math necessary to run the best upsampling algorithms.

 

I thought that AVX512 was better than AVX2 and asked Miska about your advice. That is the answer:

 

There was my mistake in thinking. The EC modulators can use AVX512 but it leads to clock throttling. The performance is similar to the i9-9900K that I use. But with filters, AVX512 can give a better result.

 

With which CPU, which mainboard and which HQPlayer settings did you test 621 and what were the limits?

[jabbr]

18 minutes ago, OAudio said:

 

If I have this right I don't think I use PCIe this way at all (must check again ) but it puts me more at the mercy of the motherboard implementation of the relevant ports for electrical signal integrity of the ports. Still the clocking is good for the reasons mentioned earlier.

I don’t think most people need to, but with NMVE storage, PCIe lanes are running out.

 

Yes, for example my NIC has extraordinarily good clocks, and there are USB cards that can be optimized for audio, etc ... again there are different ways to optimize things 

[jabbr]

1 minute ago, StreamFidelity said:

 

I thought that AVX512 was better than AVX2 and asked Miska about your advice. That is the answer:

 

There was my mistake in thinking. The EC modulators can use AVX512 but it leads to clock throttling. The performance is similar to the i9-9900K that I use. But with filters, AVX512 can give a better result.

 

With which CPU, which mainboard and which HQPlayer settings did you test 621 and what were the limits?

Yes, the AVX512 throttles the CPU which is why the base clock speeds are important. Jussi’s development machine and my workstation are essentially the same. 
 

It probably ends up being close to a toss up, the i9-9900k or new i9-10900k have a faster clock, but no AVX512 so end up equally fast. That’s why I think for most people stick with i9-10900k for new builds. I am not recommending that people get Mellanox ConnectX-5 100Gbe NICs but would point out that their clocks are only bettered by the ConnectX-6 series! In the same vein, new PCIe-4.0 capable motherboards are going to require extremely low jitter clocks/chips and overall design. I’m not sure this matters for audio but if you want to be “extreme” there is more than one way to do that  

[Peter Avgeris]

I am really speechless of this nice conversation. 

Nenon, Yes, I am open for full height chassis. But let's first check what we need to focus on, which MoBo/CPU(s)/PCH architecture. 

 

Regarding my personal experience, specially for Euphony, single or dual pc setup, I haven't found a reason to go for high computational power. I have never liked oversampling with HQPlayer and too many CPU cores. Also, I have not got positive results with multi-threading & turbo boost. Maybe I am totally wrong but my own results have been straightforward so far, however I am always willing to listen for new ideas and opinions of people that are more experienced than me. 

[Nsxturbo]

Interesting discussion.

 

Peter,

Since you have the Sage WS621E Motherboard and the scalable Xeon’s on the way, I’d love for you to at least make the CPU heat sinks with the standard HDPlex diameter heat-pipes. I too would be in for a full PCIe card height case with extra room for the extended ATX board. But honestly, I think custom bending heat-pipe for a  currently source-able DYI case is not that hard for most of us to accomplish on our own if needed.

Just the “two cents” of a current Sage WS621 owner🙃

Todd

[jabbr]

16 hours ago, StreamFidelity said:

 

With which CPU, which mainboard and which HQPlayer settings did you test 621 and what were the limits?

 

I modified a Dell T640 Poweredge server (hmm its C620 but no matter really), using dual Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz (10 cores each) and gobs of ECC RAM (96 Gb), RTX 2080 Ti GPU and a Mellanox ConnectX-4 100Gbe NIC --- the HQPlayer performance was awful! Its a great machine ... and crazy bargain ... which I use for other stuff...not audio. I'm not saying it has bad "SQ" rather that it doesn't have a high enough clock rate to do HQPLayer well.

 

Basically, HQPlayer is limited by the dual core CPU clock rate ... that's the bottom line, perhaps a tad, but really only a tad by RAM (I did that experiment on my W-2245 workstation) and negligible by NIC ... the Mellanox does RDRAM, offloads etc....

 

Bottom line is CPU clock.

 

AVX512: sure its great and I got the W-2245 so I would have that but unfortunately it clamps the clock rate to base -- I measured that personally (results on my thread) and at the end of the day, the reduced clock rate is offset by the improved processing capabilities. I selected the Xeon W-2245 (using @Miska's advice) specifically because it has a high base clock rate -- 3.9 Ghz -- even though it can burst to 4.7 Ghz ... I did hit a thermal limit using HQPlayer Pro and tuned my fans as a result.... now you can overclock an i9-10900k, and overclocking would give better performance than W-2245 but for my machine I wanted the PCIe lanes so tradeoff. I would have got a more expensive Xeon CPU with more cores but they *all* have lower base clock rates.

[rickca]

20 hours ago, OAudio said:

1. The clock subsystem. Wow this is where Intel have really helped out in the design. 

• A single oscillator on the PCH generates the CPU, RAM, DMI, PCIe, SATA, USB and Network (for on-chip N/W) clock domains signals,
• All these domains and subsystems now locked to each via the single clock - the PPLs generating the clocks for the domains all reference the same master,
• The master clock is even an integer multiple of the USB clock frequency ! so exceptional for good timing of on board USB interfaces for music if you want,
• Finally the power system for the master clock is very different to consumer motherboards, the design effectively carves out almost all of the clock subsystem, (its PLLs and transmission buffers) away from most of the motherboards power rails and other systems.

Can you please provide a link to the source of this information?  Thanks in advance.

[OAudio]

1 minute ago, rickca said:

Can you please provide a link to the source of this information?  Thanks in advance.

Its spread across multiple sources and I tend not to compile as I read, but its out there if you look at c621 architecture information sources and board reviews.

 

The summary above covers the key points regards the clock subsystem.

 

OAudio

[Peter Avgeris]

Summing it up...

 

Up to now, we have three options in total:

1. ASUS C621e SAGE (https://www.asus.com/Commercial-Servers-Workstations/WS-C621E-SAGE/)
   This motherboard is loaded with LGA3647 socket to handle scalable Xeon family of processors. It seems that many guys already have this MoBo. Copper bricks for heat dissipation do not exist and need to be designed & manufactured from scratch.
2. ASUS ProArt Z490-CREATOR 10G (https://www.asus.com/Motherboards/ProArt-Z490-CREATOR-10G/)
   This motherboard is loaded with the new LGA1200 socket for 10th gen i7/i5/i3 Intel processors. Coolers for LGA1200 are the same as in LGA115x stock copper bricks are compatible
3. ASUS Pro WS W480-ACE (https://www.asus.com/Motherboards/Pro-WS-W480-ACE/)
   This motherboard is also loaded with LGA1200 socket for the newest Xeon W-Series processors. No need for new parts
4. ASUS Pro WS X570-ACE (https://www.asus.com/Motherboards/Pro-WS-X570-ACE/)
   This is AMD AM4 socket for newest AMD processors with support for PCIe 4

From all the above, only the 1st C621e MoBo is not compatible with the HDPlex H5 chassis. It is the one that I am expecting to receive pretty soon. It is so interesting that a few guys here have access to W480 motherboard. If I was to go from scratch, I would select this one, but with audio sometimes things get really crazy.

I will have the chance to evaluate this motherboard with only the basics so as to compare apples to apples. Without excessive HQPlayer processing, it would be nice to compare my best (so far) sounding Xeon system(s) to this motherboard. If anybody living in EU has a W480 along with its own Xeon (any type) and could part its beloved MoBo for around a month (including 2-way shipment), I would be thankful. I intend to compare everything I have access to, so as to know all of us which might be the next step in digital streaming. And of course X570 is of equal (if not higher) interest as well...

[StreamFidelity]

51 minutes ago, jabbr said:

AVX512: sure its great and I got the W-2245 so I would have that but unfortunately it clamps the clock rate to base

 

Thank you for this honest assessment. It saves me from an expensive bad buy. The W-2245 has the smaller socket: FCLGA2066 and a very low temperature limit: 59 °C.

 

Apart from the utopian price, I have only found the Gold Processor 6250 with the socket FCLGA3647 and 3.90GHz to use HQPlayer with the EC modulators. That remains a utopia for now. 😆

[rickca]

57 minutes ago, OAudio said:

its out there if you look

I tried confirming this information by googling data sheets and motherboard reviews before I asked for your source.  So far, no luck.  

[jabbr]

10 minutes ago, StreamFidelity said:

and a very low temperature limit: 59 °C.

No it sings along at 60 C ... temp limit is 88

[Peter Avgeris]

21 hours ago, Nenon said:

A lot of speculations of what works and what does not. Having tested and retested some of the stuff multiple times, I disagree with quite a few things said here about what works better and what is worse. But I don't think that's the right thread for these discussions and don't really have the time to go into hot debates. I suggest we focus on how the Hdplex can be modded for more versatile DIY usage with some of the specific motherboards and CPUs mentioned here. It's obvious that @Peter Avgeris's intentions are not to design a new case from scratch. He seems open to that idea, but the topic here is about the Hdplex H5 case. 

 

Our common friend from Germany has shipped him his Asus Sage motherboard and dual CPU after discussing with me. Big thanks to him for that! He is also willing to deal with receiving any components from Peter and shipping outside of the EU (i.e. to the US for instance). Plus shipping from Germany to the US is not that expensive. 

 

Can we go back on track?

Peter - I had four suggestions given your feedback. Which ones do you think are doable? Appreciate your help with this.

 

Thank you!

Yes! I am open to do a full-height version, it is not something that difficult. One CPU can connect to the right side and the other CPU to the other side, there is plenty of free space for copper heat pipes for the second CPU. HDPlex H5 chassis is 100% perfect for ASUS C621 SAGE MoBo equipped with 2x CPUs up to 95W each one. I think that it is reasonable to focus on this project first. It is doable, you won't need to break the bank, it is nice 'n cozy, low-profile height. Now, if you want to connect HDPlex 800w P/S unit, then, OK, do something externally, Larry has the ATX chassis sockets. I think that this needs to be No.1 priority.

Yes, shipping to and from Germany is nothing important. Shipping to and from USA is troublesome and very problematic. At least we have this alternative that works fine.

All your suggestions seem to be absolutely doable. The one that changes things dramatically is the full height, because I need to order Junior (https://www.kuehlkoerper.de/en/) or Fischer (https://www.fischerelektronik.de/) Heat Sinks. These factories have tremendous support and can do all the grooves and whole metal work in general, threads opening, etc on the sinks in order to accommodate copper pipes of required diameter, back plates, front plate. Just name it and you have it, I explained you from the beginning this is sooo easy for me to do. The most difficult aspect of all (trust me, I have big experience in collaborating with people!) is to join different people's opinion and try to mix the ingredients so as to end up with the most convenient, beautiful and cost effective solution for all of us.

[jabbr]

11 minutes ago, StreamFidelity said:

Gold Processor 6250

 

Oh nice! though 8 cores so ... W-2245 is a comparative bargain and at that price I'd need to think really long and really hard about why not cluster my machines using a really fast interconnect, and not for audio but if I wanted to build an AVX512 HPC cluster ... and then there is always GPU so I'm interested in seeing what applications deploy this chip!

[The Computer Audiophile]

1 hour ago, Peter Avgeris said:

ASUS ProArt Z490-CREATOR 10G

+1

[Downtheline]

anyone else hoping for Supermicro SPM-TPF, or other narrow 3647 socket motherboards?

 

https://www.supermicro.com/en/products/motherboard/X11SPM-TPF

[Nenon]

1 hour ago, Peter Avgeris said:

Yes! I am open to do a full-height version, it is not something that difficult. One CPU can connect to the right side and the other CPU to the other side, there is plenty of free space for copper heat pipes for the second CPU. HDPlex H5 chassis is 100% perfect for ASUS C621 SAGE MoBo equipped with 2x CPUs up to 95W each one. I think that it is reasonable to focus on this project first. It is doable, you won't need to break the bank, it is nice 'n cozy, low-profile height. Now, if you want to connect HDPlex 800w P/S unit, then, OK, do something externally, Larry has the ATX chassis sockets. I think that this needs to be No.1 priority.

Yes, shipping to and from Germany is nothing important. Shipping to and from USA is troublesome and very problematic. At least we have this alternative that works fine.

All your suggestions seem to be absolutely doable. The one that changes things dramatically is the full height, because I need to order Junior (https://www.kuehlkoerper.de/en/) or Fischer (https://www.fischerelektronik.de/) Heat Sinks. These factories have tremendous support and can do all the grooves and whole metal work in general, threads opening, etc on the sinks in order to accommodate copper pipes of required diameter, back plates, front plate. Just name it and you have it, I explained you from the beginning this is sooo easy for me to do. The most difficult aspect of all (trust me, I have big experience in collaborating with people!) is to join different people's opinion and try to mix the ingredients so as to end up with the most convenient, beautiful and cost effective solution for all of us.

 

How about the idea of bolting together two H5 heatsinks on top of each other? Wouldn't that be easier and cheaper? 

[austinpop]

@Peter Avgeris

 

Thanks for starting this fantastic thread! What I am loving is the level of civility and collaboration from disparate groups in this forum. This, along with the other DIY threads started by @Nenon and @StreamFidelity are required reading for cutting-edge DIY computer audio. You guys have already surpassed my rudimentary DIY inclinations, but I hope to watch and learn.

 

We're split in this community between people who value different things:

• Software: HQPlayer and CPU-intensive upsampling vs. extremely low-CPU utilization, native sample rate servers, like Roon
• Topology: single server/player vs. distributed server+endpoint
• Design philosophy for servers:  extremely powerful CPUs (frequency, no. of CPUs/cores, large cache) vs. low-power units — even for low CPU utilization use cases.

I know my own experience is leading me down the path similar to @Nenon: 

• native sample rate extremely low-CPU utilization software
• single server/player
• extremely powerful CPUs,

but others want something different. People hold strong views around these approaches, and I'm already seeing some debate/skepticism creeping into the thread.

 

My sincere advice would be that you as the OP focus the design goals around common ground between multiple camps, while leaving flexibility around mobos, CPUs, and PSUs to satisfy all camps.

 

I would hate to see this turn into a debate thread.

[jabbr]

2 hours ago, Peter Avgeris said:

Summing it up...

 

...

1. ASUS ProArt Z490-CREATOR 10G (https://www.asus.com/Motherboards/ProArt-Z490-CREATOR-10G/)
   This motherboard is loaded with the new LGA1200 socket for 10th gen i7/i5/i3 Intel processors. Coolers for LGA1200 are the same as in LGA115x stock copper bricks are compatible

 

I like this option a lot and recommend this for new builds which don't need lots of PCIe lanes ...

 

 

Quote

1. ASUS Pro WS W480-ACE (https://www.asus.com/Motherboards/Pro-WS-W480-ACE/)
   This motherboard is also loaded with LGA1200 socket for the newest Xeon W-Series processors. No need for new parts

 

I think I confused the picture here. The LGA1200 socket is for the Xeon W 1200 series, not the W-2200 series which is LGA2066. I don't recommend Xeon W 1200 series, instead I'd consider the ASUS WS 422 (https://www.asus.com/us/Motherboards/WS-C422-PRO-SE/)

 

Sorry for the confusion on my part.

 

Quote

1. ASUS Pro WS X570-ACE (https://www.asus.com/Motherboards/Pro-WS-X570-ACE/)
   This is AMD AM4 socket for newest AMD processors with support for PCIe 4

 

Yes but AMD does not support AVX512 nor the highest CPU clock rates so I'd personally stick with i9-10900k

Quote

From all the above, only the 1st C621e MoBo is not compatible with the HDPlex H5 chassis. It is the one that I am expecting to receive pretty soon. It is so interesting that a few guys here have access to W480 motherboard. If I was to go from scratch, I would select this one, but with audio sometimes things get really crazy.

I will have the chance to evaluate this motherboard with only the basics so as to compare apples to apples. Without excessive HQPlayer processing, it would be nice to compare my best (so far) sounding Xeon system(s) to this motherboard. If anybody living in EU has a W480 along with its own Xeon (any type) and could part its beloved MoBo for around a month (including 2-way shipment), I would be thankful. I intend to compare everything I have access to, so as to know all of us which might be the next step in digital streaming. And of course X570 is of equal (if not higher) interest as well...

 

There are different ways to do things, I am offering my own experience and optimizations. Even though I don't personally have a Z490 that's what I would recommend in general and no doubt folks who have one will be very pleased, at least based on what I know  

 

[bobfa]

Thanks to @austinpop I found this thread.  I am in the process of thinking about a new single PC server.  I have an H5 Case that my Xeon is in now.  I was looking at what @austinpop was using and have skimmed @Nenon is doing.    

 

I have been surfing some of the silent PC builds, etc.  I have also looked at the big open air faceless CPU coolers. Maybe they are “art” like some of the Tube Gear where the tubes are on display!  (No derail intended, just a side note.)

 

This is looking like a must-follow thread.