Page History

Table of Contents

Overview

We assume the site will be buying 15 Compute GPU Nodes.  Things like the power and ethernet cables are based on this assumption.

Head Node

$19,772

• Dell PowerEdge R750 with 2 Xeon Silver 4310, 2.1GHz, 12core, 2U, 12-bay, $12,828 Dell PowerEdge R750   Dell PowerEdge R750
• Memory
  • 2 x 32GB, DDR4-3200 RDIMM, 2 x $158 = $366 SK-HYNIX HMA84GR7DJR4N-XN
• Drives
  • 4 x 960TB 2.5" SSDs (OS RAID1), 4 x $770 = $3,080 Samsung PM863 (3 drives for the array and one cold spare in the drawer)
  • 10 x 16TB 3.5" SAS HDDs (DATA RAID6), 10 x $330 = $3,300 https://www.newegg.com/seagate-exos-x18-st16000nm004j-16tb/p/1Z4-002P-022Y4?Item=1Z4-002P-022Y4 (9 drives for the array and one cold spare in the drawer)
• Drive Caddies
  • 12 x 3.5" drive caddies for PowerEdge R750, WorkDone 12-pack $198 https://www.amazon.com/WorkDone-4-Pack-Compatible-Installation-Screwdriver/dp/B08VF2Y7XG/?th=1

...

The test node purchased for NRAO was a Compute GPU Node, common and this is what we are recommending for all the other clusters.

Compute GPU Node, common

(760W.  Dell recommends 208V to get the full capacity of the Power Supply)

I unknowingly mistakenly bought power supplies for the NRAO test system with C20 recepticals (2400W) instead of c14 recepticals (1400W).

$16,747

•  Dell PowerEdge R750 with 2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U, 8-bay,  2400W1400W, $8,700  Dell PowerEdge R750    Dell PowerEdge R750
• Memory
  • 8 x 32GB, DDR4-3200, RDIMM, 8 x $158 = $1,264 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe Drive
  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372
• Drive Caddies (only need to order 2 12-packs total)
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack, $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/
• GPUs
  • 2 x NVIDIA RTX A5000, 230W, 2 x $2,600 = $5,200 https://www.amazon.com/NVIDIA-Quadro-RTX-A5000-Graphics/dp/B098R6RKXL

Compute CPU Node, Silver

(300W

) $11,010

• 2 Xeon Silver 4309Y, 2.89GHz, 8core, 1U, 10-bay,  $6,899 Dell PowerEdge R650
•  Memory
  • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

Compute CPU Node, Gold

(400W

) $14,663

• 2 Xeon Gold 6334, 3.6GHz, 8core, 1U, 10-bay,  $10,552 Dell PowerEdge R650
•  Memory
  • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

...

We require 208 Volt power.  Two common plugs in the United States for 208V are NEMA L21-30P for 30Amp power and IEC60309 60A 3P+PE for 60Amp power.  Each site will need to choose one of these options.

...

.

...

...

NEMA

...

NEMA L21-30P

• APC APDU9965, 8.6kW, 0U, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965

...

• Nodes + spare
  • 3-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 6-foot, C13C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 2-foot, C13/C14, qty 2

...

• Nodes + spare
  • 4-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 4-foot, C13C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 3-foot, C13/C14, qty 2

...

• One 2U SKU: KH-1922-3-100-02 2U drawer https://www.rackmountsolutions.net/kendall-howard-1922-3-100-02-2u-rack-mountable-drawer/

UPS (Optional)

(just for switch and head node)

...

• APC Smart-UPS SMT1500RM2UC 120V $1,050

...

• APC Smart-UPS X SMX2200R2HVNC 208V, $3,125

Software

Since we are planning to use all free and open-source software (RockyLinux and HTCondor) there isn't a need for software costs.

...