...
ANSWER: Ville compiled a version of casa without libatlas. I have installed it here /lustre/aoc/projects/vlass/krowe/casa-CAS-13375-2 and it exhibits the cpuset issue.
QUESTION: Does the number of threads per process change based on the cpuset?
ANSWER: Yes. Perhaps this is actually what is generating different images.
- 8 even and 0 odd cores: max of 23 threads per python3 process
- 2 even and 6 odd cores: max of 19 threads per python3 process
- 3 even and 5 odd cores: max of 17 threads per python3 process
QUESTION: Since CASA-5 doesn't suffer this cpuset problem, what if we run CASA-6 with CASA-5's FFT libraries?
ANSWER: CASA-6 doesn't suffer the cpuset issue. I copied /home/casa/packages/RHEL7/release/casa-pipeline-release-5.6.1-8.el7/lib/libfftw3* to my installation of casa-6.1.1-10-pipeline-2020.1.0.36/lib, overwriting the libfft files that were there and then tested my installation of casa-6.1.1-10-pipeline-2020.1.0.36 and was unable to produce the cpuset issue.