...
CPUs at CHTC are noticibly slower than CPUs at NRAO. For example, their set of c20xx machines (e20{03..18}) each have two Intel Xeon Silver 4114 2.20GHz processors and 0.5TB to 1TB of memory, while their large memory machines (mem3, mem2001, mem2002) each have four Intel Xeon E7-4820 v4 2.00GHz processors and 2TB to 4TB of memory. Yet, tasks seem to run faster at CHTC than at NRAO. Why?
I ran a small data set test with full parameters at CHTC that copied cfcache from /staging to local disk and step05 took only 16.7 hours instead of the 56.8 hours it had taken using cfcache on /staging.
CASA-6
Small Data Set
Small data set VLASS1.2.sb36484946.eb36542800.58574.4235612037_ptgfix_split_smaller.ms with full parameters, using cfcache from local disk.
| Step | NRAO (run06) | NRAO/CHTC () | NRAO/AWS () |
|---|---|---|---|
| 01 | 1.0 | ||
| 05 | 4.8 | ||
| 06 | 1.0 | ||
| 07 | 1.2 | ||
| 15 | 4.0 | ||
| 16 | 0.8 | ||
| 23 | 3.0 | ||
| 24 | 1.3 | ||
| Total | 17.1 |
CASA-5
Large Data Set
Large Large data set VLASS1.2.sb36491855.eb36574404.58585.53016267361_datacolumn.ms with full parameters and copying , using cfcache to local disk at CHTC.I can't run the NRAO/CHTC job until RHEL-7.8.0.0 (devhost004) gets xvfb-runfrom local disk.
| Step | NRAO (steps-all-parallel9) | NRAO/CHTC (steps-all-parallel17) | NRAO/AWS (steps-all-parallel16) |
|---|---|---|---|
| 01 | 9.4 | 36.343 | 8.49 |
| 05 | 60.2 | 53171.8526 | 67.53 |
| 06 | 24 | 27.9 | 24.87.6 |
| 07 | 11.8 | 8.4 | 511.32 |
| 15 | 55.2 | 161.0 | 61.6 |
| 16 | 6.1 | 4.0 | 5.7 |
| 23 | 230.8 | 140.1 | |
| 24 | 46 | 54.4 | |
| Total | 443.5 | 373.9 |
Small Data Set
Small data set test.ms with full parameters and not copying cfcache to local disk at CHTC using the 8k (right) cfcache and copying the cfcache to local disk at CHTC, using cfcache from local disk.
| Step | NRAO (steps-all-parallel21) | NRAO/CHTC (steps-all-parallel19) | NRAO/AWS (steps-all-parallel20) |
|---|---|---|---|
| 01 | 1.2.0 | 1.2 | 1.2.0 |
| 05 | 155.35 | 13.2 | 65.73 |
| 06 | 41.56 | 2.22 | 1.08 |
| 07 | 21.13 | 1.4 | 21.34 |
| 15 | 145.64 | 12.06 | 5.02 |
| 16 | 1.60 | 1.0 | 1.70 |
| 23 | 176.86 | 13.9 | 76.14 |
| 24 | 73.05 | 7.2 | 3.4 |
| Total | 6426.91 | 52.1 | 3125.27 |
Wallclock time from start to finish for the small data set (test.ms)
- NRAO: 65.2 not much time waiting for nodes. This is expected.26.5
- NRAO/CHTC: 111.5 so this job spent about as much time waiting for nodes as running
- NRAO/AWS: 33.0 not much time waiting for nodes. This is expected.27.3