Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND nTH

285759 krowe 20 0 4866784 2.2g 69296 S 93.7 0.4 0:35.54 python3 39
285760 krowe 20 0 4964980 2.2g 69300 S 155.6 0.4 0:38.87 python3 39
285761 krowe 20 0 4932320 2.2g 69304 S 75.2 0.4 0:35.55 python3 39
285762 krowe 20 0 4997844 2.2g 69280 S 159.9 0.4 0:38.60 python3 39
285763 krowe 20 0 4997856 2.2g 69300 S 70.5 0.4 0:35.34 python3 39
285764 krowe 20 0 4866776 2.2g 69300 S 160.9 0.4 0:39.06 python3 39
285765 krowe 20 0 4374072 2.2g 68888 S 82.1 0.4 0:14.55 python3 24
285909 krowe 20 0 311248 23936 11232 S 0.0 0.0 0:00.05 Xvfb 17
285911 krowe 20 0 311248 23932 11232 S 0.0 0.0 0:00.04 Xvfb 17
285913 krowe 20 0 311248 23932 11232 S 0.0 0.0 0:00.04 Xvfb 17
285914 krowe 20 0 311248 23932 11232 S 0.0 0.0 0:00.04 Xvfb 17
285916 krowe 20 0 311248 23936 11232 S 0.0 0.0 0:00.04 Xvfb 17
285917 krowe 20 0 311248 23932 11232 S 0.0 0.0 0:00.03 Xvfb 17
285922 krowe 20 0 311248 23936 11232 S 0.0 0.0 0:00.04 Xvfb 17
285758 krowe 20 0 1983876 205980 71444 S 0.3 0.0 0:04.69 python3 12
285756 krowe 20 0 105276 3468 2480 S 0.0 0.0 0:00.17 mpirun 2
277642 krowe 20 0 178140 2760 1200 S 0.0 0.0 0:00.09 sshd 1
277647 krowe 20 0 121564 4180 1900 S 0.0 0.0 0:00.28 bash 1
285720 krowe 20 0 115244 1492 1268 S 0.0 0.0 0:00.00 imagin+ 1
285729 krowe 20 0 115240 1528 1280 S 0.0 0.0 0:00.00 xvfb-r+ 1
285741 krowe 20 0 101684 8072 3448 S 0.0 0.0 0:00.06 Xvfb 1



QUESTION: run with OMP_NUM_THREADS=1 explicitly set in the script

ANSWER: setting OMP_NUM_THREADS=1 in the imaging.sh script still produces different images with different cpusets.  I.e. no change.  And I see OMP_NUM_THREADS=1 in /proc/PID/environ on the parent CASA process as well as all the python child processes even when I don't set OMP_NUM_THREADS in the script or in my .bashrc.


QUESTION: run with deconvolver=mtmfs and nterms=1

ANSWSER: This exhibits the cpuset issue.


QUESTION: run with deconvolver=multiscale

ANSWSER: This exhibits the cpuset issue.  It does not show the cpuset issue with the casa-5 fft libraries.


QUESTION: Do AMD processors show the same cpuset issue?

ANSWER: Yes.  While AMD numbers its cores differently, a 8-0 job produces different image than a 2-6 job.


QUESTION:

Jira
serverDMS JIRA
serverIdeb2e750b-a83a-387e-8345-36eee8a98f01
keyCAS-13313
with the compile options as similar to casa-5 as possible.

ANSWER: This new build of CASA with "as close a practical to duplicating the CASA 5 build flags" exhibits the cpuset issue.


QUESTION: Is libatlas to blame for the cpuset issue?

ANSWER: Ville compiled a version of casa without libatlas. I have installed it here /lustre/aoc/projects/vlass/krowe/casa-CAS-13375-2 and it exhibits the cpuset issue.


QUESTION: Does the number of threads per process change based on the cpuset?

ANSWER: Yes.  Perhaps this is actually what is generating different images.

  • 8 even and 0 odd cores: max of 23 threads per python3 process
  • 2 even and 6 odd cores: max of 19 threads per python3 process
  • 3 even and 5 odd cores: max of 17 threads per python3 process


QUESTION: Since CASA-5 doesn't suffer this cpuset problem, what if we run CASA-6 with CASA-5's FFT libraries?

ANSWER: CASA-6 doesn't suffer the cpuset issue.  I copied /home/casa/packages/RHEL7/release/casa-pipeline-release-5.6.1-8.el7/lib/libfftw3* to my installation of casa-6.1.1-10-pipeline-2020.1.0.36/lib, overwriting the libfft files that were there and then tested my installation of casa-6.1.1-10-pipeline-2020.1.0.36 and was unable to produce the cpuset issue.


QUESTION: Does CASA-6 honor OMP_NUM_THREADS?

ANSSER: No and yes.  If you set OMP_NUM_THREADS=0 the python3 processes spawned by mpicasa (e.g. -n 8 spawns eight python3 processes) will have more than 0 threads.  But with cpuse=0,2,4,6,8,10,12,14 and -n 8 there seems to be a pattern forming after OMP_NUM_THREADS=4

  • OMP_NUM_THREADS=0 max threads seen for a python3 process is 23
  • OMP_NUM_THREADS=1 max threads seen for a python3 process is 23
  • OMP_NUM_THREADS=2 max threads seen for a python3 process is 23
  • OMP_NUM_THREADS=4 max threads seen for a python3 process is 24
  • OMP_NUM_THREADS=8 max threads seen for a python3 process is 28
  • OMP_NUM_THREADS=16 max threads seen for a python3 process is 36


QUESTION: What determines thread count in CASA-5?

ANSWER:

  • Not cpuset
  • Not the -n argument to mpicasa.
  • OMP_NUM_THREADS but it seems to only increase the thread count if is is greater than 1.
    • OMP_NUM_THREADS=1 max threads seen for a python process is 10
    • OMP_NUM_THREADS=2 max threads seen for a python process is 11
    • OMP_NUM_THREADS=4 max threads seen for a python process is 15
    • OMP_NUM_THREADS=8 max threads seen for a python process is 27
    • OMP_NUM_THREADS=16 max threads seen for a python process is 51


QUESTION: does the number of threads per process vary based on cpuset size?

ANSWER: Yes

  • cpuset=0,2,4,6 -n 5 max threads for python3 is 15
  • cpuset=0,2,4,6,8,10,12,14 -n 5 max threads for python3 is 23
  • no cpuset defined (24 cores) -n 5 max threads for python3 is 39


QUESTION: are the system fft libraries (/usr/lib64/libfft*) used by casa-6 or casa-5?

ANSWER: No.  I used ls --time=atime --sort=time -r /usr/lib64/libfft* to see if the access time changes after running CASA.  They did not.  But when running my own installation of CASA6, the fft libraries in that installation did update their access time.


QUESTION: does adding system.resources.cores: 1 to ~/.casarc change threading in CASA-6?

ANSWER: No.  and images are still different depending on cpuset.

  • cpuset=0,2,4,6,8,10,12,14 -n 8 produces max threads of 23
  • cpuset=0,1,2,3,5,7,9,11 -n 8 produces max threads of 19