Numbers are in hours
CPUs at CHTC are noticibly slower than CPUs at NRAO. For example, their set of c20xx machines (e20{03..18}) each have two Intel Xeon Silver 4114 2.20GHz processors and 0.5TB to 1TB of memory, while their large memory machines (mem3, mem2001, mem2002) each have four Intel Xeon E7-4820 v4 2.00GHz processors and 2TB to 4TB of memory. Yet, tasks seem to run faster at CHTC than at NRAO. Why?
I ran a small data set test with full parameters at CHTC that copied cfcache from /staging to local disk and step05 took only 16.7 hours instead of the 56.8 hours it had taken using cfcache on /staging.
Large data set VLASS1.2.sb36491855.eb36574404.58585.53016267361_datacolumn.ms with full parameters and copying cfcache to local disk at CHTC.
| Step | NRAO (steps-all-parallel9) | NRAO/CHTC (steps-all-parallel17) | NRAO/AWS (steps-all-parallel16) |
|---|---|---|---|
| 01 | 9.4 | ||
| 05 | 60.2 | ||
| 06 | 24 | ||
| 07 | 11.8 | ||
| 15 | 55.2 | ||
| 16 | 6.1 | ||
| 23 | 230.8 | ||
| 24 | 46 | ||
| Total | 443.5 |
Small data set test.ms with full parameters and not copying cfcache to local disk at CHTC using the 8k (right) cfcache and copying the cfcache to local disk at CHTC.
| Step | NRAO (steps-all-parallel21) | NRAO/CHTC (steps-all-parallel19) | NRAO/AWS (steps-all-parallel20) |
|---|---|---|---|
| 01 | 1.0 | 1.2 | 1.3 |
| 05 | 13.2 | ||
| 06 | 2.2 | ||
| 07 | 1.4 | ||
| 15 | 12.0 | ||
| 16 | 1.0 | ||
| 23 | 13.9 | ||
| 24 | 7.2 | ||
| Total | 52.1 |
Wallclock time from start to finish for the small data set (test.ms)
- NRAO: 65.2 not much time waiting for nodes. This is expected.
- NRAO/CHTC: 111.5 so this job spent about as much time waiting for nodes as running
- NRAO/AWS: 33.0 not much time waiting for nodes. This is expected.