VLASS calibration pipeline CASA-5 vs CASA-6

Track Brian Kent's issue.

Comparing CASA-5 and CASA-6 (casa-pipeline-validation-8) across the two different CPUs available for batch processing in NM and CV shows that the newer CPUs (E5-2640v3) run a small calibration job (6.7GB) about 1.25 times faster than the old CPUs (E5-2670) with CASA-6 performing slower in every case.  There was no significant run-time difference between NM and CV for similar hardware and software.

Full, serial pipeline with small dataset

RHEL7 - 6.7GB dataset with NM Lustre-2.5.5

RHEL7 - 6.7GB dataset after NM upgrade Lustre-2.10.8 and CV results copied from last test

"*" Means it completed with tclean() errors

Just serial hifv_importdata() with large dataset

RHEL7 - 350GB dataset with NM Lustre-2.5.5

You can see that running just hifv_importdata() on a larger data set (350GB) shows that nmpost nodes run about 2% to 10% faster than similar cvpost nodes with CASA-6 performing slower in every case.

RHEL7 - 350GB dataset with NM Lustre-2.10.8

Both serial hifv_importdata() and hifv_hanning().

RHEL6 - with NM Lustre-2.5.5

RHEL7 - with NM Lustre-2.5.5

Full parallel pipeline with -n 8

RHEL7

"*" After 14 days of running the setjy task, and using Felipe's profiling metrics, I canceled the job.

Full parallel pipeline with -n 9

RHEL7

"*" Means it completed with pbcor errors