Page History

Table of Contents

Overview

We assume the site will be buying 15 Compute GPU Nodes.  Things like the power and ethernet cables are based on this assumption.

Head Node

$19,772

• Dell PowerEdge R750 with Head Host 2 Xeon Silver 4310, 2.1GHz, 12core, 2U, 12-bay, $7,750 $12,828 Dell PowerEdge R750   Dell PowerEdge R750
• Memory
  • 2 x 32GB, DDR4-3200 RDIMM, 2 x $158 = $366 SK-HYNIX HMA84GR7DJR4N-XN
• Drives
  • 3 4 x 960TB 2.5" SSDs (OS RAID1), 3 4 x $770 = $2$3,310 080 Samsung PM863 (3 drives for the array and one cold spare in the drawer)
  • 10 9 x 16TB 3.5" SAS HDDs (DATA RAID6), 9 10 x $330 = $2$3,970 300 https://www.newegg.com/seagate-exos-x18-st16000nm004j-16tb/p/1Z4-002P-022Y4?Item=1Z4-002P-022Y4 (9 drives for the array and one cold spare in the drawer)
• Drive Caddies
  • 12 x 3.5" drive caddies for PowerEdge R750, WorkDone 12-pack $198 https://www.amazon.com/WorkDone-4-Pack-Compatible-Installation-Screwdriver/dp/B08VF2Y7XG/?th=1

Compute Node options

The test node purchased for NRAO was a Compute GPU Node, common and this is what we are recommending for all the other clusters.

Compute GPU Node, common

760W.  Dell recommends 208V to get the full capacity of the Power Supply)

I mistakenly bought power supplies for the NRAO test system with C20 recepticals (2400W) instead of c14 recepticals (1400W).

$16,747

•  Dell PowerEdge R750 with 2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U, 8-bay,  1400W, $8,700  Dell PowerEdge R750    Dell PowerEdge R750
• Memory
  • 8
  CPU Host, Gold (400W) $14,663
  
  • 2 Xeon Gold 6334, 3.6GHz, 8core, 1U, 10-bay,  $10,552 Dell PowerEdge R650
  •  Memory16 x 32GB, DDR4-3200, RDIMM, 16 8 x $158 = $2$1,528 264 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe Drive
  drive
  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372
• Drive Caddies (only need to order 2 12-packs total)
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/
  Compute CPU Host, Silver (300W)
• GPUs
  • 2 x NVIDIA RTX A5000, 230W, 2 x $2,600 = $5,200 https://www.amazon.com/NVIDIA-Quadro-RTX-A5000-Graphics/dp/B098R6RKXL

Compute CPU Node, Silver

300W

$11,010

• 2 Xeon Silver 4309Y, 2.89GHz, 8core, 1U, 10-bay,  $6,899 Dell PowerEdge R650
•  Memory
  • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

Compute CPU Node, Gold

400W

$14,663

• 2 Xeon Gold 6334, 3.6GHz, 8core, 1U, 10 GPU Host (760W Dell recommends 208V to get the full capacity of the Power Supply) $15,194
   2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U, 8-bay,  $7$10,147 552 Dell PowerEdge R750R650
•  Memory
  Memory
  • 8 16 x 32GB, DDR4-3200, RDIMM, 8 16 x $158 = $1$2,264 528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive
  Drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies (only need to order 2 or 3 12-packs)
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/
  • GPUs
    • 2 x NVIDIA RTX A5000, 230W, 2 x $2,600 = $5,200 https://www.amazon.com/NVIDIA-Quadro-RTX-A5000-Graphics/dp/B098R6RKXL

Power Strip Options

We require 208 Volt power.  Two common plugs in the United States for 208V are NEMA L21-30P for 30Amp power and IEC60309 60A 3P+PE for 60Amp power.  Each site will need to choose one of these options.

NEMA L21-30P

• Comput GPU Host, Dense
  
  • Advanced HPC
    
    • Tyan B7109F77DV14HR-2T-N 4U GPU Server (Realfast)
      
      • Dual socket 2nd Gen Xeon Scalable Processor Family
      • 8 Double-wide PCIe x16 slots for GPU card deployment
      • https://www.advancedhpc.com/collections/high-performance-servers/products/tyan-b7109f77dv14hr-2t-n-4u-gpu-server
    • Quanta D52G-4U 4U GPU server
      
      • Up to 8 NVIDIA® Tesla® V100 with NVLink™ support up to 300GB/s GPU to GPU communication
      • Up to 10 dual-width 300 Watt GPU or 16 single-width 75 Watt GPU support
      • https://www.advancedhpc.com/collections/high-performance-servers/products/quanta-d52g-4u-4u-gpu-server
  • Dell
    • Dell EMC DSS 8440
      • I have a spec sheet for this unit but can't find it on Dell's web site.  We may have to call them to get quotes.
      • https://www.delltechnologies.com/asset/fi-fi/products/servers/technical-support/dss8440-server-specsheet-cascade-lake.pdf
      • https://www.dell.com/en-us/blog/dell-emc-dss-8440-dynamic-machine-learning-server/
  • Other?
• Power Strips
• 120V (Rack supports 2 0U power strips on each side, or four total.  There isn't going to be that much cabling, just Ethernet, so there is certainly room for 2 and maybe up to 4 0U power strips.  If you need more than 4 power strips, the extras will have to be 2U.)
  
  • APC AP8932, 0U, 120V, 2.8kW, NEMA L5-30P input, NEMA 5-20R output, $1,450 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-2G-switched-0U-30A-100V-to-120V-24-NEMA-5-20R-sockets/P-AP8932
    
  • APC AP7902B, 2U, 120V, 2.8kW, NEMA L5-30P input, NEMA 5-20R output, $1,375 https://www.apc.com/shop/us/en/products/Rack-PDU-Switched-2U-30A-120V-16-5-20/P-AP7902B
    
208VAPC APDU9965, 8.6kW, 0U, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965

IEC60309 60A 3P+PE

• APC APDU9967, 17kW, 0U, Input is IEC60309 60A 3P+PE 3Phase, Outputs are 42x C13/C15 and 6x C19/C21, $3,775 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-17-3kW-208V-42-C13-and-C15-6-C19-and-C21-sockets/P-APDU9967
• UPS (just for switch and head node)
  
  • 120V
    • APC Smart-UPS SMT1500RM2UC 120V $1,050
      
  • 208V
    • APC Smart-UPS X SMX2200R2HVNC 208V, $3,125
      

Switch

• Cisco Catalyst C9300X-48TX
  

  • 48 Data, 48x 10G Multigigabit

  • 100M, 1G, 2.5G, 5G, or 10 Gbps
  • Switching capacity 2,000 Gbps
  • Nine optional Modular Uplinks 100G/40G/25G/10G/1G
  • Redundant Power Supply 715 W
  • $12,000

Environment Monitor

(only need one)

• APC AP9335TH, Temperature and Humidity Sensor, length is 3.9m, $190 https://www.apc.com/shop/us/en/products/APC-Temperature-Humidity-Sensor/P-AP9335TH

Rackmount KVM

• StarTech RKCONS1901, Rackmount KVM console, 1U, $990 https://www.cdw.com/product/startech.com-rackmount-kvm-console-1u-19-lcd-vga-kvm-drawer-w-cables-usb/5103418?pfm=srh

Rack Cabinet

• APC NetShelter SX AR3100, $1,875 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Black-1991H-x-600W-x-1070D-mm/P-AR3100 

Power Cable Options

Some data centers have overhead power (Top power) and some have underfloor power (Bottom power).  Each site will need to choose one of these two options.

Top power

• Nodes + spare
  • 3-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 6-foot, C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 2-foot, C13/C14, qty 2

Bottom power

• Nodes + spare
  • 4-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 4-foot, C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 3-foot, C13/C14, qty 2

Ethernet Cables

...

• 10-foot, Cat-6a, qty 7
• 7-foot, Cat-6a, qty 8
• 5-foot, Cat-6a, qty 4

Blank Panels

• One 4U, SKU#: 102-1825 from https://www.racksolutions.com/filler-panels.html
• Two 2U, SKU#: 102-1823 from https://www.racksolutions.com/filler-panels.html

2U Rack Drawer

• One 2U SKU: KH-1922-3-100-02 2U drawer https://www.rackmountsolutions.net/kendall-howard-1922-3-100-02-2u-rack-mountable-drawer/

UPS (Optional)

just for switch and head node

• APC Smart-UPS X SMX2200R2HVNC 208V, $3,125

Software

Since we are planning to use all free and open-source software (RockyLinux and HTCondor) there isn't a need for software costs.