It can be tricky to properly specify script file and research software output file path names. Here are a few guidelines, to help you successfully get work done, and properly logged, in Wharton’s HPCC.
Job Script Output
The most straightforward way to collect job script output (both output and error) is … to do nothing! By default, output from jobs will be saved in the location that you ran the job from (where you were when you typed ‘qsub’), and will be saved in files named: ‘JOB_NAME.oJOB_ID’, where JOB_NAME == the name of the job (‘-N JOB_NAME’ qsub option, or ‘#$ -N JOB_NAME’ line in the job script, or JOB_NAME == the name of the script, if no ‘-N’ option was specified. The JOB_ID is the JobID of the job when it was submitted. If you run an array job, the output file will also have .TASK_ID tacked onto the end. Error files will be identical, except for the ‘o’, which will be ‘e’ instead.
If you have a very small number of jobs, we generally recommend that you don’t change that, except to add ‘-j y’ (or ‘#$ -j y’ line in the job script), to conjoin both output and error to the same file, which is generally much easier to debug and read.
If you have a lot of jobs, or many tasks in array jobs, you will be running (and many of you do!), we recommend that you create a job output directory, maybe called ‘job_output’ (for example … it doesn’t matter what you call it). Then you can specify: ‘-o job_output/$JOB_NAME.$JOB_ID.log’, or ‘-o job_output/$JOB_NAME.$JOB_ID.$TASK_ID.log’ (for array jobs), and all the output will be written into the job_output directory, in files based off of JOB_NAME, JOB_ID, and TASK_ID (if it’s an array job).
Here’s a best practice job script for you to use:
#!/bin/bash #$ -N ANY_NAME_CHANGE_ME #$ -o job_output/$JOB_NAME-$JOB_ID.log ## comment the above, and uncomment the two (2) below for an array job ##$ -t 1-10 ##$ -o job_output/$JOB_NAME-$JOB_ID-$TASK_ID.log #$ -j y Rscript --vanilla MY_R_SCRIPT.R
Command Line
If you’re adding those options on the command line, instead of in a script, you’ll need to add single quotes around the -o option, like:
qsub -N ANY_NAME_CHANGE_ME -o 'job_output/$JOB_NAME-$JOB_ID.log' -j y -b y 'Rscript --vanilla MY_R_SCRIPT.R'
Or for an array job:
qsub -N ANY_NAME_CHANGE_ME -t 1-10 -o 'job_output/$JOB_NAME-$JOB_ID-$TASK_ID.log' -j y -b y 'Rscript --vanilla MY_R_SCRIPT.R'
Research Software Output
Generally, you can just let output from your research software go to standard out, and standard error (‘the screen’), and it will be written into your job script Output / Error, as detailed above.