#!/bin/sh ### ### Skeleton of a PBS script for submitting sequential or parallel ### jobs on the PSI batch cluster: ### ### http://www.millennium.berkeley.edu/PSI/batch.html ### ### You MUST edit this skeleton and insert the parameters and commands ### for your program. ### ### PBS is the name of the batch scheduler protocol. The batch ### scheduler is responsible for assigning jobs to nodes in a cluster. ### It tries to make sure that jobs don't fight for processing ### resources, so that they can get the best performance. Without a ### batch scheduler, you can never be sure that some other job might ### interrupt your job on a node. ### ### A PBS script is a standard *nix shell script, with special ### commands for the PBS batch scheduler. Lines beginning with "#PBS" ### are NOT comments; they are PBS commands. You can comment them out ### by putting "### " in front of them. ### ### Once you've written the script, you can submit the job using ### the qsub command. See, for example, ### ### http://www.clusterresources.com/wiki/doku.php?id=torque:2.1_job_submission ### ### For example, if your script is called "pbs.sh", run the following: ### ### qsub pbs.sh ### ### Note that the PBS script itself only runs once, on one compute ### node. It is used to start either a process (which may optionally ### create threads on that node), or to call gexec or mpirun in order ### to start multiple MPI processes (which is only for MPI programs). ### ### Set the job name ### YOU MUST CHANGE THIS #PBS -N myprogram ### Declare myprogram non-rerunable ### LEAVE THIS ALONE #PBS -r n ### Set the queue to "batch", the only available queue. ### This is only an issue for systems with multiple queues. #PBS -q batch ### Specify the number of cpus for your job. This example will run on ### 8 cpus using 2 nodes with 4 processes per node. You MUST specify ### some number of nodes. Some portions of some clusters, such as ### older nodes in the PSI BATCH cluster, support Myrinet GM, which is ### a faster interface for doing internode communication with MPI ### programs. If you want to run your MPI program with GM, you should ### add one of the following flags (probably just the "gm" flag) to ### ensure that you are assigned nodes with Myrinet cards: ### ### gm ### gm_pci64b ### gm_pcixd ### ### For example, to run on 4 processes on each of 2 nodes with Myrinet ### cards you write: ### ### #PBS -l nodes=2:ppn=4:gm ### ### If you don't care about GM, do the following: ### ### #PBS -l nodes=2:ppn=4 ### ### You should avoid requesting GM unless you are using MPI and are ### concerned about communication performance. In particular, don't ### request GM for debugging, because it steals the GM nodes from the ### people who might need them for performance. ### ### If you are using threads and not MPI, then ppn can be used to ### request a node with a certain number of CPU's / cores. You should ### set ppn equal to the number of cores / CPU's on that node; ### otherwise, the batch scheduler may allocate other jobs to that ### node and thus mess up your timings. ppn has NOTHING to do with ### the number of threads that your process may create. ### ### If no nodes exist in the system with that number of processors per ### node, qsub will fail right away with an error message like this: ### ### qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes ### #PBS -l nodes=2:ppn=4 ### You can tell PBS how much memory you expect your job will use per ### node. You can give it in gigabytes (for example, 1g) or megabytes ### (for example, 1024m). PBS may ignore this value, but it's still a ### good idea to pick a reasonable upper bound. Make sure that it's ### not more memory than the node actually has! #PBS -l mem=1g ### You can override the default 1 hour real-world time limit: ### ### -l walltime=HH:MM:SS ### ### Be nice, OK? Try to keep your jobs under a few hours. There ### are probably hard limits (something like 10 days). If your job ### takes a lot longer than you thought it would, you should kill it ### so as not to take away resources from other people. ### ### Note: If your job exceeds the time limit, it will be killed, which ### means that all your results may be lost. However, the queue may ### schedule jobs first if they request a shorter walltime. This ### should encourage you to estimate your runtime as accurately as ### possible. ### ### Here we schedule the job to run for a maximum of one hour. ### #PBS -l walltime=1:00:00 ### Change to the working directory, from which processes are ### launched. By default Torque launches processes from your home ### directory. Jobs should only be run from /home, /project, or ### /work, since Torque returns results via NFS. /work is much ### preferable as it avoids overloading EECS home directories. ### ### Note: do NOT comment out this line! cd $PBS_O_WORKDIR ### Run some informational commands. You can comment these out for ### production runs, but they are handy for debugging. echo Working directory is $PBS_O_WORKDIR echo Running on host `hostname` echo Time is `date` echo Directory is `pwd` echo This jobs runs on the following processors: echo `cat $PBS_NODEFILE` ### Define number of processors in this batch script, which can be ### handy for invoking mpirun. You can comment out the "echo" line. NPROCS=`wc -l < $PBS_NODEFILE` echo This job has allocated $NPROCS cpus ### ### There are a number of options for launching processes. Pick exactly ### one of these options. If you're not running MPI, you should just run ### the program as you would normally: $PBS_O_WORKDIR/yourprogram ### ### 1. Use gexec to run a program on the assigned processors. ### $GEXEC_SVRS is automatically defined for you. ### ### gexec -n 0 myprogram ### ### 2. Run a parallel MPI executable using "mpirun". ### ### mpirun -v -machinefile $PBS_NODEFILE -np $NPROCS mympiprogram ### ### 3. If you're running on only one node, run your executable directly: ### ### /some/path/to/a/binary ###