Page History

Table of Contents

Overview

We assume the site will be buying 15 Compute GPU Nodes.  Things like the power and ethernet cables are based on this assumption.

Head Node

$19,772

• Dell PowerEdge R750 with Head Host 2 Xeon Silver 4310, 2.1GHz, 12core, 2U, 12-bay, $7,750 $12,828 Dell PowerEdge R750   Dell PowerEdge R750
• Memory
  • 2 x 32GB, DDR4-3200 RDIMM, 2 x $158 = $366 SK-HYNIX HMA84GR7DJR4N-XN
• Drives
  • 3 4 x 960TB 2.5" SSDs (OS RAID1), 3 4 x $770 = $2$3,310 080 Samsung PM863 (3 drives for the array and one cold spare in the drawer)
  • 10 9 x 16TB 3.5" SAS HDDs (DATA RAID6), 9 10 x $328 $330 = $2,952 Seagate Exos X16 ST16000NM002G$3,300 https://www.newegg.com/seagate-exos-x18-st16000nm004j-16tb/p/1Z4-002P-022Y4?Item=1Z4-002P-022Y4 (9 drives for the array and one cold spare in the drawer)
• Drive Caddies
  • 12 x 3.5" drive caddies with 2.5" adapters for PowerEdge R750, WorkDone 612-pack , 2 x $138 = $276 $198 https://www.amazon.com/WorkDone-4-Pack-Compatible-Installation-Screwdriver/dp/B08TV42J7FB08VF2Y7XG/?th=1

Compute Node options

The test node purchased for NRAO was a Compute GPU Node, common and this is what we are recommending for all the other clusters.

Compute GPU Node, common

760W.  Dell recommends 208V to get the full capacity of the Power Supply)

I mistakenly bought power supplies for the NRAO test system with C20 recepticals (2400W) instead of c14 recepticals (1400W).

$16,747

•  Dell PowerEdge R750 with 2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U, 8-bay,  1400W, $8,700  Dell PowerEdge R750    Dell PowerEdge R750
• MemoryCompute CPU Host (Gold)
  
  • 2 Xeon Gold 6334, 3.6GHz, 8core, 1U, 10-bay,  $10,552 Dell PowerEdge R650
   Memory
  • 8 x 32GB, DDR4-3200, RDIMM, 8 x $158 = $1,264 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
• NVMe Drive
  drive
  • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372
• Drive Caddies (only need to order 2 12-packs total)
  
  • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/
• GPUs
  Compute CPU Host (Silver)
  • 2 x NVIDIA RTX A5000, 230W, 2 x $2,600 = $5,200 https://www.amazon.com/NVIDIA-Quadro-RTX-A5000-Graphics/dp/B098R6RKXL

  Compute CPU Node, Silver

  300W

  $11,010

  • 2 Xeon Silver 4309Y, 2.89GHz, 8core, 1U, 10-bay,  $6,899 Dell PowerEdge R650
  •  Memory
    • 8 16 x 32GB, DDR4-3200, RDIMM, 8 16 x $158 = $1$2,264 528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
  • NVMe drive

    • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

  • Drive Caddies
    
    • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

  Compute CPU Node, Gold

  400W

  $14,663

  • 2 Xeon Gold 6334, 3.6GHz GPU Host 2 Xeon Silver 4309Y, 2.89GHz, 8core, 2U1U, 810-bay,  $7$10,147 552 Dell PowerEdge R750R650
  •  Memory
    Memory
    • 16 x 32GB, DDR4-3200, RDIMM, 16 x $158 = $2,528 https://memory.net/product/hma84gr7djr4n-xn-sk-hynix-1x-32gb-ddr4-3200-rdimm-pc4-25600r-dual-rank-x4-module/
  • NVMe drive
    Drive

    • Samsung PM1733 MZWLJ7T6HALA, 7.68TB, U.2, NVMe, $1,583 https://www.cdw.com/product/samsung-pm1733-mzwlj7t6hala-ssd-7.68-tb-pcie-4.0-x4-nvme/6254372

  • Drive Caddies
    
    • 2.5" drive caddy for PowerEdge R640 (should also work with R650), WorkDone 12-pack (only need to order 2 or 3), $200 https://www.amazon.com/WorkDone-2-Pack-Hard-Drive-Caddy/dp/B07Q4T91B3/

  Power Strip Options

  We require 208 Volt power.  Two common plugs in the United States for 208V are NEMA L21-30P for 30Amp power and IEC60309 60A 3P+PE for 60Amp power.  Each site will need to choose one of these options.

  NEMA L21-30P

  • APC APDU9965, 8.6kW, 0U, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965

  IEC60309 60A 3P+PE

  • APC APDU9967, 17kW, 0U, Input is IEC60309 60A 3P+PE 3Phase, Outputs are 42x C13/C15 and 6x C19/C21, $3,775 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-17-3kW-208V-42-C13-and-C15-6-C19-and-C21-sockets/P-APDU9967 StripsUPS

  
  

  Switch

  • Cisco Catalyst C9300X-48TX
    

    • 48 Data, 48x 10G Multigigabit

    • 100M, 1G, 2.5G, 5G, or 10 Gbps
    • Switching capacity 2,000 Gbps
    • Nine optional Modular Uplinks 100G/40G/25G/10G/1G
    • Redundant Power Supply 715 W
    • $12K$12,000

  Environment Monitor

  (only need one)

  • APC AP9335TH, Temperature and Humidity Sensor, length is 3.9m, $190 https://www.apc.com/shop/us/en/products/APC-Temperature-Humidity-Sensor/P-AP9335TH

  Rackmount KVM

  • StarTech RKCONS1901, Rackmount KVM console, 1U, $990 https://www.cdw.com/product/startech.com-rackmount-kvm-console-1u-19-lcd-vga-kvm-drawer-w-cables-usb/5103418?pfm=srh

  Rack Cabinet

  • APC NetShelter SX AR3100, $1,875 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Black-1991H-x-600W-x-1070D-mm/P-AR3100 

  Power Cable Options

  Some data centers have overhead power (Top power) and some have underfloor power (Bottom power).  Each site will need to choose one of these two options.

  Top power

  • Nodes + spare
    • 3-foot, C13/C14, Y-splitter (wide), qty 17
  • Switch + spare
    • 6-foot, C15/C14, qty 2 NOTE these are C15 not C13
  • KVM + spare
    • 2-foot, C13/C14, qty 2

  Bottom power

  • Nodes + spare
    • 4-foot, C13/C14, Y-splitter (wide), qty 17
  • Switch + spare
    • 4-foot, C15/C14, qty 2 NOTE these are C15 not C13
  • KVM + spare
    • 3-foot, C13/C14, qty 2

  Ethernet Cables

  • 10-foot, Cat-6a, qty 7
  • 7-foot, Cat-6a, qty 8
  • 5-foot, Cat-6a, qty 4

  Blank Panels

  • One 4U, SKU#: 102-1825 from https://www.racksolutions.com/filler-panels.html
  • Two 2U, SKU#: 102-1823 from https://www.racksolutions.com/filler-panels.html

  2U Rack Drawer

  • One 2U SKU: KH-1922-3-100-02 2U drawer https://www.rackmountsolutions.net/kendall-howard-1922-3-100-02-2u-rack-mountable-drawer/

  UPS (Optional)

  just for switch and head node

  • APC Smart-UPS X SMX2200R2HVNC 208V, $3,125

  Software

  Since we are planning to use all free and open-source software (RockyLinux and HTCondor) there isn't a need for software costs.