Transition Guide: Moving from the Wharton HPC3 to PARCC Betty

A guide for Wharton/Penn researchers moving workloads, code, and data from the Wharton HPC3 cluster to the university-wide PARCC Betty cluster.

Contents

1. Job Submission: Grid Engine to Slurm
2. GUI Access: FastX becomes Open OnDemand (OOD)
3. Copying Your Current Project Data: Use Globus
4. Archiving Older Projects: A Note on Dropbox
5. Rebuilding Python Virtual Environments and R Libraries
6. Put Your Code in Git Instead of Copying It

Before You Start: What’s Changing

Because the two clusters run different operating environments, nothing moves automatically. Move data deliberately and rebuild your software environments from scratch. It’s also a good chance to leave old projects behind rather than dragging every file across.

1. Job Submission: Grid Engine (qsub/qlogin) to Slurm (sbatch/srun)

HPC3 uses Altair (Univa) Grid Engine. Betty uses Slurm. The concepts map over, but the commands and script directives are different, so you can’t reuse an HPC3 job script on Betty as-is.

Command equivalents

Script directive equivalents

Grid Engine directives use #$ at the top of the script; Slurm uses #SBATCH.

HPC3 style (demo.sh):

#!/bin/bash
#$ -N myjob
#$ -j y
#$ -l m_mem_free=6G
#$ -pe openmp 2
#$ -q short.q

module load python
python myscript.py

Betty/Slurm equivalent:

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --partition=genoa-std-mem
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=5632M
#SBATCH --time=04:00:00

module load python/3.13.12
python myscript.py

Key differences to watch for

• Partitions replace queues. Where HPC3 had hpc3.q, short.q, mem.q, and gpu.q, Betty organizes hardware into partitions, selected with --partition=. As of this writing the main partitions are:

  Partition	Hardware	Notes
  dgx-b200	NVIDIA DGX B200 GPU nodes	For AI/ML training
  b200-mig90	90GB MIG GPU instance	Medium GPU jobs
  b200-mig45	45GB MIG GPU instance	Small GPU jobs
  genoa-std-mem	AMD Genoa CPUs, standard memory	General CPU jobs
  genoa-large-mem	AMD Genoa CPUs, high memory	Jobs needing >256GB RAM

• Always request a time limit (--time=HH:MM:SS). Be realistic, but don’t leave it off.
• GPUs are requested with --gpus=N rather than by picking a special queue, but you still need the right --partition.
• For array jobs, use sbatch --array=1-100 myjob.sh instead of qsub -t 1-100, and reference $SLURM_ARRAY_TASK_ID instead of $SGE_TASK_ID.
• For job dependencies, use sbatch --dependency=afterok:<jobid> instead of -hold_jid.
• For interactive debugging: srun --partition=genoa-std-mem --ntasks=1 --cpus-per-task=2 --mem-per-cpu=5632M --time=04:00:00 --pty bash. That’s the Betty version of qrsh/qlogin.
• If you’re in multiple projects, the -P equivalent is -A. Your default project will be PI_PENNKEY-wharton.
• Don’t run real computation on the Betty login nodes. Same rule as HPC3, but Betty enforces it more strictly since login nodes are shared across the whole university.

Before you submit anything on Betty, go through your existing HPC3 .sh job scripts and rewrite the header block from #$ directives to #SBATCH directives, and swap any embedded qsub/qrsh/qstat/qdel calls for their Slurm equivalents above.

2. GUI Access: FastX becomes Open OnDemand (OOD)

On HPC3, graphical access (RStudio, MATLAB, a full desktop) went through FastX. On Betty, the equivalent is Open OnDemand (OOD), a web portal that launches interactive apps like Jupyter or a remote desktop as Slurm jobs behind the scenes.

The key difference: on HPC3 you’d open a FastX desktop and then manually run qrsh/qlogin from a terminal to grab compute resources. On Betty’s OOD, launching an interactive app is the resource request. You pick the partition, CPU/GPU count, memory, and walltime in a form, and OOD submits the Slurm job for you.

• Like FastX sessions, OOD sessions have a hard walltime and get killed if idle. Set a walltime that covers your actual work session and close the session when you’re done, so you’re not holding resources other researchers need.
• If you relied on FastX’s “Applications” menu for RStudio/MATLAB/Stata, on Betty you either load the Lmod module (module load <tool>) inside your session, or pick a preconfigured interactive app from the OOD menu if one exists.

In your first week on Betty, log into OOD once to confirm Duo and Kerberos are working end-to-end, before you’re depending on it for a deadline.

3. Copying Your Current Project Data: Use Globus

For active projects, use Globus to move data from HPC3 to Betty. It’s more reliable than scp or rsync over a login node: transfers resume after interruptions and get checksummed at the end, so you know they actually finished.

Setting up Globus

1. See the PARCC Globus Data Transfer documentation.
2. In a browser, go to https://globus.org and click Log In.
3. Search for “University of Pennsylvania” as your organization and continue.
4. Authenticate with your PennKey and password (Duo if prompted).
5. (Optional) If you use Globus elsewhere, you can link identities. See the Globus’s identity linking guide.

Doing the transfer

1. In the Globus File Manager, open one panel to the “Wharton – HPC3 Home Directories” endpoint/collection and a second panel to the “PARCC Betty VAST Projects – High Assurance” endpoint/collection.
2. The first time you connect to a given collection, Globus will ask for your UPenn credentials and prompt you to approve a data-access consent. Do this once per collection.
3. Navigate to your source folder on the HPC3 in one panel and your destination folder (e.g. /vast/projects/<your-project>/) on the Betty side in the other panel.
4. Select the files/folders to move and click Start on the source panel.
5. Track progress under Activity in Globus.

Why Globus instead of scp/ rsync / rclone for this

• It resumes automatically if a transfer is interrupted (important for large research datasets).
• It verifies data integrity (checksums), rather than trusting that a raw copy succeeded.
• It doesn’t tie up your terminal or a login node for the duration of a multi-hour transfer.
• You get an email with a report when the transfer finishes.

Before deleting anything from HPC3, spot-check that files transferred via Globus match in size and count on the Betty side. parcc_du.py <path> and parcc_quota.py are useful for seeing what actually landed.

4. Archiving Older Projects: A Note on Dropbox

If you’d rather move old, inactive projects off the cluster instead of migrating them to Betty, Dropbox works fine for cold storage. One carryover from HPC3:

• rclone is the practical way to push data from a cluster shell to Dropbox (or Box, Google Drive, S3). Configure a remote once with rclone config, choose Dropbox and complete the OAuth flow, then archive a project with something like:

  rclone copy -u -v --transfers 4 some_project dropbox:HPC3/Archive/some_project --progress

• Alternatively, for modest amounts of data, you can download the project to your laptop first and upload it to Dropbox through the normal web interface or desktop app on your own machine, with no cluster-side Dropbox integration needed.

Before archiving, strip out anything you don’t need long-term: scratch files, old job logs, Python virtualenvs, duplicate copies of raw data. Archives are easier to live with when they aren’t just an unfiltered dump of your project folder.

5. Rebuilding Python Virtual Environments and R Libraries

Don’t try to copy your HPC3 venv/conda env/R library folders over to Betty. They were built against HPC3’s OS and CPUs and will break on Betty. Rebuild environments from a manifest instead.

Python

• On the HPC3 (while you still have access), capture what you have:

   pip freeze > requirements.txt
   # or if you used conda:
   conda env export --no-builds > environment.yml

• On Betty, load the Python/Conda tooling via modules:

   module load miniconda3

• Rebuild the environment fresh:

   pip install -r requirements.txt
   # or for conda
   conda create -n myproject python=3.13
   conda activate myproject
   

• Don’t pip install --user into the system Python on Betty; PARCC’s own guidance discourages it. Keep every project in its own named environment.
• If you use MPI or GPU-linked packages (PyTorch with CUDA, mpi4py), reinstall them against Betty’s CUDA and MPI versions rather than assuming the HPC3-built wheel will work. The GPU hardware and CUDA versions are different.

R

• On HPC3, get a list of installed packages before you leave:

   installed.packages()[, "Package"]
   # Better: if you used renv, just keep your renv.lock file

• On Betty, load the R module:

   module load arch/zen4 
   module load r/4.5.2

• If your project already uses renv, this is the easiest path: copy over just the renv.lock file (via Globus or git, not the whole library folder) and run:

   renv::restore()

from inside the project directory on Betty. This rebuilds exactly the package versions you had, compiled fresh for Betty.

• If you don’t already use renv, now’s a good time to start. It turns “what packages do I need” into a single lockfile you can move between systems, instead of a folder of compiled binaries.
• For packages with system-library dependencies (e.g. sf, rJava, anything linking to GDAL or JAGS), check module avail on Betty for the matching system modules before compiling from source. Module names and versions may differ from HPC3.

For every active project, produce a requirements.txt or environment.yml (Python) or renv.lock (R) before you lose convenient access to HPC3, so you have a manifest to rebuild from instead of reverse-engineering what was installed.

6. Put Your Code in Git Instead of Copying It

A straight copy via Globus makes sense for data. For your code, use the migration as an excuse to put it in a git repository: GitHub, GitLab, or a bare repo on shared project storage.

Why bother, when you could just copy a folder?

• A folder copy only has today’s version. A git repo has every version you ever committed, which is what you actually want when a result changes and you need to figure out why.
• It’s a real backup that’s independent of either cluster. If a copy step fails, or you find a bug in code you already “migrated,” your code isn’t only living on university-managed storage.
• The same repo works on HPC3, Betty, your laptop, or wherever you end up next: git clone instead of coordinating another file transfer.
• Commit your requirements.txt, environment.yml, or renv.lock alongside the code, and setting up the project on a new machine becomes git clone plus a rebuild from the manifest.
• Collaboration gets easier too, which matters more now that PARCC is university-wide. Collaborators outside Wharton can pull your code without needing access to Wharton storage.

A reasonable order of operations:

1. Pick a small-to-medium active project as a pilot.
2. From HPC3 (or your laptop, if you already have a local copy), initialize a repo if one doesn’t exist:

   cd myproject
   git init
   git add .
   git commit -m "Initial commit before PARCC migration"

3. Push it to a remote (GitHub/GitLab/Penn-hosted git service).
4. On Betty, git clone the repo instead of copying the code folder via Globus.
5. Keep large data files out of git (use .gitignore); those still move via Globus as described in Section 3. Git is for code and configs, not multi-gigabyte datasets.

At minimum, get your analysis and pipeline code into git before or during the move. Don’t wait until after everything’s copied over to Betty to think about version control.

Migration Checklist

• Confirm Betty account is active and you can kinit + ssh login.betty.parcc.upenn.edu
• Run srun hostname to confirm Slurm access works
• Rewrite active HPC3 qsub scripts as Slurm sbatch scripts (Section 1)
• Log into Open OnDemand once to confirm GUI/interactive access works (Section 2)
• Use Globus to move active project data from HPC3 to the right Betty storage path (Section 3)
• Decide which old projects get archived to Dropbox (via rclone) instead of migrated (Section 4)
• Export requirements.txt / environment.yml / renv.lock from HPC3 before losing access, and rebuild fresh on Betty (Section 5)
• Move current, active code into a git repository rather than copying the folder as-is (Section 6)
• Verify storage quotas on Betty with parcc_quota.py
• Confirm with your PI which partition(s)/accounts you should be using for jobs

Getting Help

• Wharton Research & Analytics (HPC3 questions, migration coordination): research-computing@wharton.upenn.edu
• PARCC support: via the PARCC support page/contact form: https://parcc.upenn.edu/support/
• PARCC documentation: https://docs.parcc.upenn.edu/