How to use SGE
with Matlab
One issue that comes up very often are system load issues related to matlab.
Matlab will happily use extra cpus or cpu-cores on the system if/when available.
I would suggest that you use the -singleCompThread option when batching matlab jobs - as otherwise, matlab will try running its process in parallel using all the system’s CPU cores … and if you have another matlab process running it will try to do the same (ditto for 3rd and so on) and you can quickly exceed the system’s resources, while these processes will likely be swapped out (of memory) more often than necessary and will translate into longer processing times.
You can also limit or control matlab’s parallel nature with the matlabpool function:
matlabpool open <num_threads>
when doing so, you must also give SGE (Sun Grid Engine) some consideration regarding multi-threading in order for SGE to properly manage it’s resources … and this is accomplished with parallel environments. Most SGE queues at the BIC have some parallel environments defined and one way to find out would be with the following commands, e.g. :
$ qconf -sq all.q | grep pe_list pe_list make all.pe $ qconf -sp all.pe pe_name all.pe slots 48 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true allocation_rule $pe_slots control_slaves FALSE job_is_first_task TRUE urgency_slots min accounting_summary FALSE
without getting into too much detail, please contact bicadmin if in doubt.
to give you an example, supposing that you used ‘matlabpool open 4’ within your matlab script, you would then submit this to all.q with:
qsub -q all.q -pe all.pe 4 ...