Page History

Table of Contents

Overview

We assume the site will be buying 15 Compute GPU Nodes.  Things like the power and ethernet cables are based on this assumption.

Head Node

$13$19,594772

• Dell PowerEdge R750 with 2 Xeon Silver 4310, 2.1GHz, 12core, 2U, 12-bay, $7$12,750 828 Dell PowerEdge R750   Dell PowerEdge R750
• Memory
  • 2 x 32GB, DDR4-3200 RDIMM, 2 x $158 = $366 SK-HYNIX HMA84GR7DJR4N-XN
• Drives
  • 3 4 x 960TB 2.5" SSDs (OS RAID1), 3 4 x $770 = $2$3,310 080 Samsung PM863
  9
  • (3 drives for the array and one cold spare in the drawer)
  • 10 x 16TB 3.5" SAS HDDs (DATA RAID6),
  9
  • 10 x $330 =
  $2
  • $3,
  970
  • 300 https://www.newegg.com/seagate-exos-x18-st16000nm004j-16tb/p/1Z4-002P-022Y4?Item=1Z4-002P-022Y4 (9 drives for the array and one cold spare in the drawer)
• Drive Caddies
  • 12 x 3.5" drive caddies for PowerEdge R750, WorkDone 12-pack $198 https://www.amazon.com/WorkDone-4-Pack-Compatible-Installation-Screwdriver/dp/B08VF2Y7XG/?th=1

...

The test node purchased for NRAO was a Compute GPU Node, common and this is what we are recommending for all the other clusters.

Compute GPU Node, common

(760W.  Dell recommends 208V to get the full capacity of the Power Supply)

I unknowingly mistakenly bought power supplies for the NRAO test system with C20 recepticals (2400W) instead of c14 recepticals .(1400W).

$15$16,194747

•  Dell PowerEdge R750 with 2  2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U, 8-bay,  $71400W, $8,147700  Dell PowerEdge R750    Dell PowerEdge R750
• Memory
  • 8 x 32GB, DDR4-3200, RDIMM, 8 x $158 = $1,264 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe Drive
  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372
• Drive Caddies (only need to order 2 or 3 12-packs total)
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack, $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/
• GPUs
  • 2 x NVIDIA RTX A5000, 230W, 2 x $2,600 = $5,200 https://www.amazon.com/NVIDIA-Quadro-RTX-A5000-Graphics/dp/B098R6RKXL

Compute CPU Node, Silver

(300W

) $11,010

• 2 Xeon Silver 4309Y, 2.89GHz, 8core, 1U, 10-bay,  $6,899 Dell PowerEdge R650
•  Memory
  • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

Compute CPU Node, Gold

(400W

) $14,663

• 2 Xeon Gold 6334, 3.6GHz, 8core, 1U, 10-bay,  $10,552 Dell PowerEdge R650
•  Memory
  • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe drive

  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

• Drive Caddies
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

Power

...

Strip Options

We require 208 Volt power.  Two common plugs in the United States for 208V are NEMA L21-30P for 30Amp power and IEC60309 60A 3P+PE for 60Amp power.  Each site will need to choose one of these options.

NEMA L21-30P

• APC
• 120V (Rack supports 2 0U power strips on each side, or four total.  There isn't going to be that much cabling, just Ethernet, so there is certainly room for 2 and maybe up to 4 0U power strips.  If you need more than 4 power strips, the extras will have to be 2U.)
  
  • APC AP8932, 0U, 120V, 2.8kW, NEMA L5-30P input, NEMA 5-20R output, $1,450 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-2G-switched-0U-30A-100V-to-120V-24-NEMA-5-20R-sockets/P-AP8932
    
  • APC AP7902B, 2U, 120V, 2.8kW, NEMA L5-30P input, NEMA 5-20R output, $1,375 https://www.apc.com/shop/us/en/products/Rack-PDU-Switched-2U-30A-120V-16-5-20/P-AP7902B
    
• 208VAPC APDU9965, 8.6kW, 0U, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965

IEC60309 60A 3P+PE

• APC APDU9967, 17kW, 0U, Input is IEC60309 60A 3P+PE 3Phase, Outputs are 42x C13/C15 and 6x C19/C21, $3,775 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-17-3kW-208V-42-C13-and-C15-6-C19-and-C21-sockets/P-APDU9967

Switch

• Cisco Catalyst C9300X-48TX
  

  • 48 Data, 48x 10G Multigigabit

  • 100M, 1G, 2.5G, 5G, or 10 Gbps
  • Switching capacity 2,000 Gbps
  • Nine optional Modular Uplinks 100G/40G/25G/10G/1G
  • Redundant Power Supply 715 W
  • $12,000

...

• APC NetShelter SX AR3100, $1,875 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Black-1991H-x-600W-x-1070D-mm/P-AR3100 

Cables

Power Cable Options

Some data centers have overhead power (Top power) and some have underfloor power (Bottom power).  Each site will need to choose one of these two options.

Top power

• Nodes + spare
  • 3-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 6-foot, C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 2-foot, C13/C14, qty 2

Bottom power

• Nodes + spare
  • 4-foot, C13/C14, Y-splitter (wide), qty 17
• Switch + spare
  • 4-foot, C15/C14, qty 2 NOTE these are C15 not C13
• KVM + spare
  • 3-foot, C13/C14, qty 2

Ethernet Cables

• 10-foot, Cat-6a, qty 7
• 7-foot, Cat-6a, qty 8
• 5-foot, Cat-6a, qty 4

Blank Panels

• One 4U, SKU#: 102-1825 from https://www.racksolutions.com/filler-panels.html
• Two 2U, SKU#: 102-1823 from https://www.racksolutions.com/filler-panels.html

2U Rack Drawer

• One 2U SKU: KH-1922-3-100-02 2U drawer https://www.rackmountsolutions.net/kendall-howard-1922-3-100-02-2u-rack-mountable-drawer/
• 3-meter, Cat-6a, RJ-45 ethernet cables
various power cords, Y-splitter, wide enough for R750

UPS (Optional)

(just for switch and head node)

...

• APC Smart-UPS SMT1500RM2UC 120V $1,050

...

• APC Smart-UPS X SMX2200R2HVNC 208V, $3,125

Software

Since we are planning to use all free and open-source software (RockyLinux and HTCondor) there isn't a need for software costs.

...