...
ANSWSER: This exhibits the cpuset issue. It does not show the cpuset issue with the casa-5 fft libraries.
QUESTION: Do AMD processors show the same cpuset issue?
...
ANSWER: CASA-6 doesn't suffer the cpuset issue. I copied /home/casa/packages/RHEL7/release/casa-pipeline-release-5.6.1-8.el7/lib/libfftw3* to my installation of casa-6.1.1-10-pipeline-2020.1.0.36/lib, overwriting the libfft files that were there and then tested my installation of casa-6.1.1-10-pipeline-2020.1.0.36 and was unable to produce the cpuset issue.
QUESTION: Does CASA-6 honor OMP_NUM_THREADS?
ANSSER: No and yes. If you set OMP_NUM_THREADS=0 the python3 processes spawned by mpicasa (e.g. -n 8 spawns eight python3 processes) will have more than 0 threads. But with cpuse=0,2,4,6,8,10,12,14 and -n 8 there seems to be a pattern forming after OMP_NUM_THREADS=4
- OMP_NUM_THREADS=0 max threads seen for a python3 process is 23
- OMP_NUM_THREADS=1 max threads seen for a python3 process is 23
- OMP_NUM_THREADS=2 max threads seen for a python3 process is 23
- OMP_NUM_THREADS=4 max threads seen for a python3 process is 24
- OMP_NUM_THREADS=8 max threads seen for a python3 process is 28
- OMP_NUM_THREADS=16 max threads seen for a python3 process is 36
QUESTION: What determines thread count in CASA-5?
ANSWER:
- Not cpuset
- Not the -n argument to mpicasa.
- OMP_NUM_THREADS but it seems to only increase the thread count if is is greater than 1.
- OMP_NUM_THREADS=1 max threads seen for a python process is 10
- OMP_NUM_THREADS=2 max threads seen for a python process is 11
- OMP_NUM_THREADS=4 max threads seen for a python process is 15
- OMP_NUM_THREADS=8 max threads seen for a python process is 27
- OMP_NUM_THREADS=16 max threads seen for a python process is 51
QUESTION: does the number of threads per process vary based on cpuset size?
ANSWER: Yes
- cpuset=0,2,4,6 -n 5 max threads for python3 is 15
- cpuset=0,2,4,6,8,10,12,14 -n 5 max threads for python3 is 23
- no cpuset defined (24 cores) -n 5 max threads for python3 is 39
QUESTION: are the system fft libraries (/usr/lib64/libfft*) used by casa-6 or casa-5?
ANSWER: No. I used ls --time=atime --sort=time -r /usr/lib64/libfft* to see if the access time changes after running CASA. They did not. But when running my own installation of CASA6, the fft libraries in that installation did update their access time.
QUESTION: does adding system.resources.cores: 1 to ~/.casarc change threading in CASA-6?
ANSWER: No. and images are still different depending on cpuset.
- cpuset=0,2,4,6,8,10,12,14 -n 8 produces max threads of 23
- cpuset=0,1,2,3,5,7,9,11 -n 8 produces max threads of 19