...
- Cabinet Rack: Doors front and rear locking with mesh. Width: 19". Height: 42U is most common. Depth: 42" is most common. Rack must support at least 2,000 lbs static load
- Great Lakes GL840ES-2442-B-MS, $3,900 This is what we usually get https://greatcabinets.com/product/es-ms/
- APC NetShelter SX AR3100, $1,875 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Black-1991H-x-600W-x-1070D-mm/P-AR3100
- APC NetShelter SX AR3100SP, designed for re-shipping after equipped, $2,550 https://www.apc.com/shop/us/en/products/APC-NetShelter-SX-Server-Rack-Enclosure-42U-Shock-Packaging-2000-lbs-Black-1991H-x-600W-x-1070D-mm/P-AR3100SP
- Power Strips: How many Power Strips? What plug? What voltage? This may very across sites. What if the site has two power sources?
- 208V = APC APDU9965, 8.6kW, zeroU, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965
- Need 2. For 30 nodes, assuming 400W each we will need 2 of these power strips (30 * 400W = 12,000W)
- 208V = APC APDU9967, 17kW, zeroU, Input is IEC60309 60A 3P+PE 3Phase, Outputs are 42x C13/C15 and 6x C19/C21, $3,775 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-17-3kW-208V-42-C13-and-C15-6-C19-and-C21-sockets/P-APDU9967
- Need 1. For 30 nodes, assuming 400W each we will need only 1 of these power strips (30 * 400W = 12,000W)
- 120V = APC AP8932, 2.8kW, zeroU, Input is NEMA L5-30P, Outputs are 24x NEMA 5-20R, $1,450 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-2G-switched-0U-30A-100V-to-120V-24-NEMA-5-20R-sockets/P-AP8932
- Need 5. For 30 nodes, assuming 400W each we will need 5 of these power strips (30 * 400W = 12,000W)
- Stagger startups on PDU
- 208V = APC APDU9965, 8.6kW, zeroU, Input is NEMA L21-30P 3phase, Outputs are 21x C13/C15 and 3x C19/C21, $2,100 https://www.apc.com/shop/us/en/products/APC-Rack-PDU-9000-switched-0U-8-6kW-208V-21-C13-and-C15-3-C19-and-C21-sockets/P-APDU9965
- UPS: for just the head node and switch? This may depend on the voltage of the PDUs.
- 208V = APC Smart-UPS X SMX2200R2HVNC, 2200VA, 2U, Input is IEC-C20, Outputs are 8x C13 and 1x C19, comes with network card, $3,125 https://www.apc.com/shop/us/en/products/APC-Smart-UPS-X-Line-Interactive-2200VA-Rack-tower-2U-208V-230V-8x-C13-1x-C19-IEC-Network-card-Extended-runtime-Rail-kit-included/P-SMX2200R2HVNC
- 120V = APC Smart-UPS SMT1500RM2UC, 1500VA, 2U, Input is NEMA 5-15P, Outputs are 6x NEMA 5-15R, $1,050 https://www.apc.com/shop/us/en/categories/power/uninterruptible-power-supply-ups-/network-and-server/smart-ups/N-1h89ykeZ11ai1p3
- Optional UPS Network Card 3 AP9640 $420 https://www.apc.com/shop/us/en/products/UPS-Network-Management-Card-3/P-AP9640
- Switch: Cisco Catalyst C9300X-48TX
48 Data, 48x 10G Multigigabit
- 100M, 1G, 2.5G, 5G, or 10 Gbps
- Switching capacity 2,000 Gbps
- Nine optional Modular Uplinks 100G/40G/25G/10G/1G
- Redundant Power Supply 715 W
- $12K
- Environmental Monitoring: Add-on to the APC PDU
- APC AP9335TH, Temperature and Humidity Sensor, length is 3.9m, $190 https://www.apc.com/shop/us/en/products/APC-Temperature-Humidity-Sensor/P-AP9335TH
- KVM: rackmount, not remote, and patch cables
- StarTech RKCONS1901, Rackmount KVM console, 1U, $990 https://www.cdw.com/product/startech.com-rackmount-kvm-console-1u-19-lcd-vga-kvm-drawer-w-cables-usb/5103418?pfm=srh
- Ethernet cables:
- Power cables: single or Y cables depending on number and types of PDUs and number of power sources.
- Head Node: lots of disk and RAID
- iDRAC: Ask CIS what they recommend
- Memory: at least 32GB of RAM to help cache the OS image. 64GB would be even better.
- Storage mdRAID, ZFS, Btrfs, RAID card? Do we want both boot and data arrays to be the same type?
- OS/OSimages, RAID1 with or without spare? (3 disks), about 1TB
- OS: We have been making 40GB partitions for / for over a decade and that looks to still work with RHEL8.
- Swap: 0 or 8GB at most
- /export/home/<hostnmae>: services and diskless_images
- Working data/software (remote institution and nrao), RAID6 w/spare or RAID7 (9 disks), about 72TB
- An SE imaging input data size is about 10GB per job
- We need maybe 20TB+ of total space or more so maybe 60TB/2
- Carve into two partitions (NRAO data and NRAO software, remote institution data and remote institution software) each partition has data and software directories.
- OS/OSimages, RAID1 with or without spare? (3 disks), about 1TB
- Networking: May need more than one port. One for internal networking to nodes and one for external Internet access.
- 30 1U nodes or 15 2U nodes or mix? NVMe drives for nodes. Swap drive?
- Dual ~3GHz, ~12Core CPUs
- 512GB or more RAM
- NVMe for scratch and swap 6TB or more
- Draws about 400W max
- GPUs: Do we get GPUs? Do we get 1U nodes with room for 1 or 2 Tesla T4 GPUs or 2U nodes with room for 1 or 2 regular GPU?
- Dell R650, dual Xeon Gold 6334 3.6GHz CPUs, 768GB RAM, 6TB - 8TB NVMe
- Dell R650, dual Xeon Silver 4309Y 2.8GHz CPUs, 768GB RAM, 6TB - 8TB NVMe
- PassMark shows the Silver to be about 85% to 95% of the Gold but the Gold is about 2.5 times the price of the Silver. For the price of 19 Gold nodes we could get 26 Silver nodes. So we could run 73% more jobs witlh Silvers but they would take maybe 15% longer to run. That looks like a win to me. (assuming 15 jobs per node).
- Cooling: Assuming a 30-node cluster, we will need about 12kW or 3.5 tons or 42,000 BTU/h of cooling.
...