...
- Cabinet Rack: Doors front and rear locking with mesh. Width: 19". Height: 42U is most common. Depth: 42" is most common. Rack must support at least 2,000 lbs static load
- Great Lakes GL840ES-2442-B-MS, $3,900 This is what we usually get https://greatcabinets.com/product/es-ms/
- APC NetShelter SX AR3100, $1,875 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Black-1991H-x-600W-x-1070D-mm/P-AR3100
- APC NetShelter SX AR3100SP, designed for re-shipping after equipped, $2,550 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Shock-Packaging-2000-lbs-Black-1991H-x-600W-x-1070D-mm/P-AR3100SP
- Power Strips: How many Power Strips? What plug? What voltage? This may very across sites. What if the site has two power sources?
- 208V = APC APDU9965, 8.6kW, zeroU, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965
- Need 2. For 30 nodes, assuming 400W each we will need 2 of these power strips (30 * 400W = 12,000W)
- 208V = APC APDU9967, 17kW, zeroU, Input is IEC60309 60A 3P+PE 3Phase, Outputs are 42x C13/C15 and 6x C19/C21, $3,775 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-17-3kW-208V-42-C13-and-C15-6-C19-and-C21-sockets/P-APDU9967
- Need 1. For 30 nodes, assuming 400W each we will need only 1 of these power strips (30 * 400W = 12,000W)
- 120V = APC AP8932, 2.8kW, zeroU, Input is NEMA L5-30P, Outputs are 24x NEMA 5-20R, $1,450 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-2G-switched-0U-30A-100V-to-120V-24-NEMA-5-20R-sockets/P-AP8932
- Need 5. For 30 nodes, assuming 400W each we will need 5 of these power strips (30 * 400W = 12,000W)
- Stagger startups on PDU
- 208V = APC APDU9965, 8.6kW, zeroU, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965
- UPS: for just the head node and switch? This may depend on the voltage of the PDUs.
- 208V = APC Smart-UPS X SMX2200R2HVNC, 2200VA, 2U, Input is IEC-C20, Outputs are 8x C13 and 1x C19, comes with network card, $3,125 https://www.apc.com/shop/us/en/products/APC-Smart-UPS-X-Line-Interactive-2200VA-Rack-tower-2U-208V-230V-8x-C13-1x-C19-IEC-Network-card-Extended-runtime-Rail-kit-included/P-SMX2200R2HVNC
- 120V = APC Smart-UPS SMT1500RM2UC, 1500VA, 2U, Input is NEMA 5-15P, Outputs are 6x NEMA 5-15R, $1,050 https://www.apc.com/shop/us/en/categories/power/uninterruptible-power-supply-ups-/network-and-server/smart-ups/N-1h89ykeZ11ai1p3
- Optional UPS Network Card 3 AP9640 $420 https://www.apc.com/shop/us/en/products/UPS-Network-Management-Card-3/P-AP9640
- Switch: Cisco Catalyst C9300X-48TX
48 Data, 48x 10G Multigigabit
- 100M, 1G, 2.5G, 5G, or 10 Gbps
- Switching capacity 2,000 Gbps
- Nine optional Modular Uplinks 100G/40G/25G/10G/1G
- Redundant Power Supply 715 W
- $12K
- Environmental Monitoring: Add-on to the APC PDU
- APC AP9335TH, Temperature and Humidity Sensor, length is 3.9m, $190 https://www.apc.com/shop/us/en/products/APC-Temperature-Humidity-Sensor/P-AP9335TH
- KVM: rackmount, not remote, and patch cables
- StarTech RKCONS1901, Rackmount KVM console, 1U, $990 https://www.cdw.com/product/startech.com-rackmount-kvm-console-1u-19-lcd-vga-kvm-drawer-w-cables-usb/5103418?pfm=srh
- Ethernet cables:
- Power cables: single or Y cables depending on number and types of PDUs and number of power sources.
- Head Node: lots of disk and RAID
- iDRAC: Ask CIS what they recommend
- Memory: at least 32GB of RAM to help cache the OS image. 64GB would be even better.
- Storage mdRAID, ZFS, Btrfs, RAID card? Do we want both boot and data arrays to be the same type?
- OS/OSimages, RAID1 with or without spare? (3 disks), about 1TB
- OS: We have been making 40GB partitions for / for over a decade and that looks to still work with RHEL8.
- Swap: 0 or 8GB at most
- /export/home/<hostnmae>: services and diskless_images
- Working data/software (remote institution and nrao), RAID6 w/spare or RAID7 (9 disks), about 72TB
- An SE imaging input data size is about 10GB per job
- We need maybe 20TB+ of total space or more so maybe 60TB/2
- Carve into two partitions (NRAO data and NRAO software, remote institution data and remote institution software) each partition has data and software directories.
- OS/OSimages, RAID1 with or without spare? (3 disks), about 1TB
- Networking: May need more than one port. One for internal networking to nodes and one for external Internet access.
- 30 1U nodes or 15 2U nodes or mix? NVMe drives for nodes. Swap drive?
- Dual ~3GHz, ~12Core CPUs
- 512GB or more RAM
- NVMe for scratch and swap 6TB or more
- Draws about 400W max
- GPUs: Do we get GPUs? Do we get 1U nodes with room for 1 or 2 Tesla T4 GPUs or 2U nodes with room for 1 or 2 regular GPU?
- Cooling: Assuming a 30-node cluster, we will need about 12kW or 3.5 tons or 42,000 BTU/h of cooling.
Shipping
- Drop ship everything to the site and assemble on site. This will require an NRAO person on site to assemble with a pre-built OS disk for the head node. I think this is too much work to do on site.
- Install DIMMs in nodes
- Install NVMe drives in nodes
- Rack everything
- Cable everything
- Configure switch
- Install/Configure OS
- Ship everything here and assemble then ship a rack-on-pallet.
- Mix the two. Ship minimal stuff here (head node, switch, couple of compute nodes, etc) and configure and drop ship most of the nodes to the site.
- Re-ship head node, switch, compute nodes to site
- Re-ship memory and drives to site
- A person from the remote site could travel to NM or CV to see the test system and get instruction.
...