Numbers are in hours
CPUs at CHTC are noticibly slower than CPUs at NRAO. For example, their set of c20xx machines (e20{03..18}) each have two Intel Xeon Silver 4114 2.20GHz processors and 0.5TB to 1TB of memory, while their large memory machines (mem3, mem2001, mem2002) each have four Intel Xeon E7-4820 v4 2.00GHz processors and 2TB to 4TB of memory.
CASA-6
Small Data Set
Small Large data set VLASS1.2.sb36491855sb36484946.eb36574404eb36542800.58585.53016267361_datacolumn58574.4235612037_ptgfix_split_smaller.ms with full parameters and copying , using cfcache to from local disk at CHTC.
| Step | NRAO ( | steps-all-parallel9run06) | NRAO/CHTC (steps-all-parallel17) | NRAO/AWS (steps-all-parallel16) | |
|---|---|---|---|---|---|
| 01 | 9.4 | 27.7 (ran at CHTC) | 12.31.0 | ||
| 05 | 60.2 | 65.94.8 | |||
| 06 | 24 | 241.40 | |||
| 07 | 11.8 | 21.41.2 | |||
| 15 | 554. | 20 | 58.9 | ||
| 16 | 60. | 18 | 7.6 | ||
| 23 | 2303. | 80 | |||
| 24 | 461.3 | ||||
| Total | 44317.1 |
CASA-5
Large Data Set
Small Large data set test VLASS1.2.sb36491855.eb36574404.58585.53016267361_datacolumn.ms with full parameters and not copying cfcache to local disk at CHTC using the 16k (wrong) cfcache, using cfcache from local disk.
| Step | NRAO (steps-all-parallel12parallel9) | NRAO/CHTC (steps-all- | parallel15parallel17) | NRAO/AWS (steps-all- | parallel14parallel16) |
|---|---|---|---|---|---|
| 01 | 19.842 | 6. | 04 | 18.9 | |
| 05 | 860.6256 | 171. | 85 | 567. | 13 |
| 06 | 3.0243 | 27.9 | 224. | 08 | |
| 07 | 211.082 | 8. | 34 | 211.2 | |
| 15 | 655.9256 | 161. | 30 | 461. | 36 |
| 16 | 6.1 | 4. | 40 | 15.7 | 1.4|
| 23 | 8.3 | 47230.85 | 140. | 31 | |
| 24 | 14.146 | 66.0 | 16.854.4 | ||
| Total | 46443.15 | 226373.8 | 39.0 |
CPUs at CHTC are noticibly slower than CPUs at NRAO. For example, their set of c20xx machines (e20{03..18}) each have two Intel Xeon Silver 4114 2.20GHz processors and 0.5TB to 1TB of memory, while their large memory machines (mem3, mem2001, mem2002) each have four Intel Xeon E7-4820 v4 2.00GHz processors and 2TB to 4TB of memory. While this can cause jobs to run slower, the biggest factor was probably using cfcache on their shared filesystem.
I ran a small data set test with full parameters at CHTC that copied cfcache from /staging to local disk and step05 took only 16.7 hours instead of the 56.8 hours it had taken using cfcache on /staging.
| 9 |
Small Data Set
Small data set test.ms with full parameters and not copying cfcache to local disk at CHTC using the 8k (right) cfcache and copying the cfcache to local disk at CHTC, using cfcache from local disk.
| Step | NRAO (steps-all-parallel21) | NRAO/CHTC (steps-all-parallel19) | NRAO/AWS (steps-all-parallel20) |
|---|---|---|---|
| 01 | 1.2.0 | 1.2 | 1.2.0 |
| 05 | 155.35 | 13.2 | 65.73 |
| 06 | 41.56 | 2.22 | 1.08 |
| 07 | 21.13 | 1.4 | 21.34 |
| 15 | 145.64 | 612.0 | 5.2 |
| 16 | 1.0 | 1.60 | 1.70 |
| 23 | 6.6 | 1713.89 | 76.14 |
| 24 | 3.5 | 7.2 | 3.4 |
| Total31.2 | 26.1 | 52.1 | 25.7 |
Wallclock time from start to finish for the small data set (test.ms)
- NRAO: 26.5
- NRAO/CHTC: 111.5 so this job spent about as much time waiting for nodes as running
- NRAO/AWS: 27.3