Matlab jobs can make significant use of the inherent multithreading capability, which speeds up elementwise computations such as those done by the sin and log functions, and computations that use the Basic Linear Algebra Subroutines (BLAS) library, such as matrix multiply. If you run a matlab job, by default it will use multithreaded libraries and try and run on all of the cores in the node it is submitted to (ie 48 cores in serial.q and inf.q, and 64 cores on the amd queues etc).
This might not matter if it was a small part of the overall execution of a job, but some jobs make considerable use of multithreaded libraries.
For example, if you were to submit ten matlab jobs to the serial.q, each job would try to use all 48 cores.
10 * 48 is > 48. The node is overloaded.
The effect is that other peoples jobs receive much less priority than they should and you may well crash the node through memory exhaustion.
If you submit Matlab jobs you MUST explicitly restrict the number of threads each job uses.
You can run jobs single threaded by launching Matlab with the -singleCompThread argument. This might be the best way to do things at first, though of course each job will take longer to complete:
matlab -singleCompThread <script
There is a command maxNumCompThreads (see help maxNumCompThreads within Matlab) that should allow you to set the maximum number of threads a process will use. I’ve not yet tested it recently - this facility is set to be withdrawn, and may already have been disabled.
Assuming that it works, you could set it to say 4 threads. You could then submit a job using the openmp parallel environment requesting 4 threads, which would limit the job to using 4 cores.
Your job script might be,
#!/bin/sh #$ -pe openmp 4 -q serial.q module add matlab/R2012b unset DISPLAY matlab << EOF addpath('~yourusername/DIR',path); maxNumCompThreads(4) MyMatlabProg quit EOF
qsub -binding linear:1 -b y "matlab -singleCompThread <script"