MPICH and Mvapich with Torque Integration

It has come up a few times on the OSC mpiexec, MPICH, and MVAPICH mail lists about tight Torque support. The following should also apply to PBS Pro and and any other MPICH derivative (MATLAB PCT, Intel MPI, etc).  This is possible and I am writing it down here for those who keep asking.

This is desirable because without Task Manager (TM) support from PBS a job on remote nodes will be spawned by ssh, and thus will be outside the view of the batch system.  When used with TM, reported memory and cputime will add up, and jobs can be killed off cleaner and remain inside respective CPU Sets/Cgroups if configured.

One used to need to use the OSC mpiexec to get good PBS/TM (task manager) support.  Mpiexec is no longer needed and does not currently work with Torque 4.x. So then what?

Enter Hydra, thank goodness MPICH finally killed of MPD, it always felt like a hack to me, and didn't support resource manager plugins.

Hydra has two sets of plugins, one set known as rmk is how Hydra gets the lists of hosts to spawn on.  Hydra builds always out the box with this support for PBS and will use it if it detects a PBS job. The other, launcher, is how Hydra spawns the jobs on remote nodes.  The default in almost all cases is ssh which is not what you want.

I never could get hydra that comes with mpich to pick up libtoque.so correctly to enable the launcher. So I needed to download it on its own.  Lucky for us Mpich provides Hydra as its own package.

One can omit --with-hydra-bss=pbs if they want to leave all the other launchers/rmks available.  If you run mpiexec.hydra -help you should find a launcher and rmk for pbs. If you have only one something went wrong.

You can also use the environment variables HYDRA_LAUNCHER=pbs HYDRA_RMK=pbs so set the desired rmk or launcher to PBS if you didn't force bss to pbs.

If everything looks good, I install the custom built Hydra binaries over the ones installed with Mpich.  In testing with a multi-node parallel the any node other than the first should have  hydra_pmi_proxy a child of pbs_mom, pstree is my favorite tool for this.