Run Unlimited Jobs with the Short Queue

The short.q job queue on the Wharton High Performance Computing Cluster (HPCC) is now available for general use. What this means is that all users may now have up to 256 more jobs simultaneously running! This is a way around the current 64-core limit. To use this new feature, one may select the short queue or specify a four-hour or less time limit per job.

An example of specifying the short queue:
$ qsub -q short.q myjob.sh

An example of specifying a time limit of four hours:
$ qsub -l h_rt=04:00:00 myjob.sh

The short queue should provide a quick turn-around for all short-running jobs while encouraging programmers to break up work into more manageable chunks. If you have jobs that take days or weeks to complete, please consider how the work may be logically divided into four-hour segments. Utilizing the short queue allows for more efficient use of the finite resources of both the cluster and of your own research time.

BIG FAT WARNING: All running jobs are prioritized via a weighted and equitable fairshare policy per project. Specifying the short queue will limit jobs to four hours after which they will be unceremoniously killed. Objects in short queue may appear larger than they seem. Ask a doctor before using short queue.

As a specialist in Linux and high-performance computing, Burris enjoys enabling faculty within The Wharton School of the University of Pennsylvania by providing effective research computing resources. Burris has been involved in research computing since 2001. Current projects find Burris working with HPC, big data, cloud computing and grid technologies. His favorite languages are Python and BASH. In his free time, he enjoys bad cinema, video editing, synthesizers and bicycling.