NVMe Hardware RAID storage with DRAM cache

[yamamoto2002]

I just built the Windows PC with hardware RAID1 NVMe boot drive with Adaptec 3258-16i/e

 

■ Why RAID1 boot drive

 

The same data is written onto two drives. If one of the drives failed, it is not need to reinstall Windows, just replace the failed drive to fresh one. If you are lucky enough, the system continues to work.

 

This provides some feel of safety BUT if you are not so lucky, or, accidentally you replace wrong drive, RAID is destroyed, it is need to reinstall Windows and restore your data from the daily backup. (I recommend to take the backup onto NAS or different computer). So backup is still necessary.

 

■ Hardware RAID and software RAID

 

There are two types of RAID exists.

• Software RAID does not require special hardware, just connect 2 or more drives onto PC and software do the RAID job. Recent years Software RAID becomes more popular.
• “Hardware RAID controller” is needed for hardware RAID setup. Two or more drives are connected onto the controller, and the controller do the RAID tasks and operating system see it as a single drive.

 

■ Performance gain from DRAM cache on the controller

 

Hardware RAID controller has DRAM cache. My 3258-16i/e has 8GB DRAM: 4GB read cache and 4GB write cache. This means, operating system disk write request smaller than 4GB are translated to

• Copy the data onto the DRAM queue on the controller
• Reports “write finished” to operating system (it is called write back operation)
• Queued data will be written onto disks

 

When sudden power loss is happened, 

• The controller continues to work with dedicated battery or super capacitor as a power source and DRAM cell refresh will be continued to keep the data alive for some time.
• Data queued on DRAM will be copied onto flush memory on the controller before backup power drained.
• On the next boot-up, DRAM data is restored from flush memory and the controller resumes RAID tasks.

Therefore 4GB of data loss does not happen (at least in theory)

 

PCIe Gen4 x8 Hardware RAID card DRAM cache performance

 

■ Tri-mode adapter

 

Tri-mode hardware RAID controller is recently introduced. It supports

• SATA drives
• SAS drives
• NVMe devices

 

Especially NVMe support is attractive, it is expected to build faster RAID than traditional SATA/SAS controllers. I chose Adaptec 3258-16i/e tri-mode hardware RAID controller.

 

■ NVMe devices used

 

I'd like to go cheap, chose M.2 NVMe,

Seagate Firecuda 530 PCIe Gen4 2TB

 

The M.2 drive is converted to U.3 drive using Micro SATA Cables M.2 to U.3 adapter https://www.amazon.com/dp/B08GCV9BG1

then connected to the controller with SlimSAS 8X to 2*U.2 NVMe Adapter https://www.amazon.com/dp/B098JBS7QD

 

I tested several other adapters and cables and found it is only setup to work properly.

 

Hardware RAID controller, super capacitor, and U.3 to M.2 adapters

 

■ Computer parts list

 

• AMD Ryzen 7950x CPU
• ASRock X670E Taichi motherboard, BIOS 1.24
• Corsair Vengeance DDR5 DRAM 5600MHz 48GB x2
• Coolermaster ML360L V2 cpu cooler
• Silverstone SETA D1 ATX case
• Seasonic Prime 850W ATX power supply
• Adaptec 3258-16i/e hardware RAID controller.
• Seagate Firecuda 530 PCIe Gen4 2TB M.2 NVMe device x2
• NVidia Titan V
• Nvidia RTX 4090 connected with Thunderbolt 4 with Razer Chroma X

 

I tried one Intel 12th gen computer and one Intel 11th gen computer and both does not work with the Adaptec card: BIOS does not start. It works with Intel Xeon E5 v4 Supermicro X10DRL-i. It does not work with AMD Threadripper 2990WX Asus Zenith Extreme.

 

 

■ Windows setup procedure

 

• On first boot-up, enter UEFI BIOS menu, on the menu list, select Adaptec 3258 setup, then Create Array → choose two NVMe drives and select RAID1 to create the logical drive.
• Run Windows 11 22H2 boot USB, surprisingly it has the device driver of the controller and it recognizes the logical drive. OS installation process is smooth.
• Logon to Windows and install Max Storage Manager and see backup power is charged or not (DRAM cache is enabled when backup power is ready), tweak controller parameters, then run disk performance benchmarks.

 

U.3 adapter SMBus EEPROM jumper settings. It is recognized as SFF-TA-1001 PCIe (U.3)

• POS 4-9 Hi(1)
• POS 5-8 Low(0)
• POS 6-7 Low(0)

 

BTW Gen-Z is now part of CXL.

 

U.3😉  Drives connected

 

Controller setup

 

RAID1 drive configuration

 

[yamamoto2002]

Another CrystalDiskMark result, with "NVMe SSD" test set.

 

Achieved the speed of theoretical limit of PCIE gen4 x8 link with 5% PCIe protocol overhead

 

[jcbenten]

Very cool...

 

Out of curiosity...what do you do with that kind of speed?  In daily use do you see delays/stutters because the SW or non-optimized HW need to "catch up"?

 

Thanks for sharing.

[yamamoto2002]

I'm working to improve WAV file read performance of PlayPcmWin, when I have a time and willingness to do, it is recreational project 

 

Before the optimization, standard fopen / fread / fclose is used, and it is saturated at ～2GB/sec by API bottleneck.

 

Using “I/O Completion Ports” asynchronous file read and AVX512 PCM bit-depth converter, 5 GB/sec WAV file read is achieved with single M.2 drive and a proof of concept code.

• https://sourceforge.net/p/playpcmwin/code/HEAD/tree/PlayPcmWin/00experiments/TestNativeSoundFileReader/
• https://sourceforge.net/p/playpcmwin/code/HEAD/tree/PlayPcmWin/WWAsmLib/
• https://sourceforge.net/p/playpcmwin/code/HEAD/tree/PlayPcmWin/WWCommonCpp/WWFileReaderMT.cpp

 

Still other part of the source code needs rewrite to improve overall throughput. In order to motivate myself, I arranged this faster storage system