Data Protection
All user data (/home) are protected by Snapshotting. Please see our Data Protection Page for details on the infrastructure as well as how to recover files.
Quotas
Wharton’s HPC3 has storage connected to each compute node to work with files. Your main area for file storage is your “home” directory:
/home/department/username or /home/department/facultypi/team/username
(sometimes abbreviated as ~, also available as the $HOME variable). You can get the full path to your home directory using the command:
echo $HOME
While the cluster has dedicated storage (in addition to WRDS data access), resources are not unlimited. Therefore, each user and department as a whole has a quota or limit on the size of files allowed. The default quotas are:
- Per Team*: 128GB (software quotas)
* RAs and Co-Authors share this storage with their sponsor. These limits can be adjusted above the defaults by special request and chargeback.
To check on your quota usage, use the following command: quota
Daily Warning E-mails
On a daily basis all users’ quotas are checked, and if they are above 96% (default) the primary user and all group members (RAs, Co-Authors) associated with the primary user are e-mailed with a warning message with details at their @wharton.upenn.edu address.
Adjusting Warning Triggers
Primary users can adjust the percentage of their quota at which they receive these messages, or to whom these messages are sent, by creating and/or editing a .quotacheck file in their home directory. The primary user and all group members (RAs, Co-Authors) will always be notified via their @wharton.upenn.edu e-mail addresses, but you can add additional users, and/or modify the warning level as below:
WARN_LEVEL 85 NOTIFY someotheruser@gmail.com,myphonenumber@myphonecarrier.com
Increasing Quotas
If you find that you need more than the default 100GB of storage space, please get in touch with Research Computing with details about your needs. This is a fee-based service, chargeable annually to a Penn budget code.
Please note: in FY 2025 (July 2024) we will move to monthly or quarterly (still negotiating) billing instead of annual.
| Size | Cost |
|---|---|
| < 1TB | $0.5/GB/year |
| 1TB-9TB | $500/TB/year |
| 10TB+ | $400/TB/year |
Transferring Files
There are two currently supported file transfer protocols for moving files on and off the HPC3:
- SSH-FTP (SSH File Transfer Protocol, not Secure FTP)
- SMB (Windows file sharing)
SSH-FTP is implemented in the command scp already installed on Mac OS X and Linux and many file transfer clients available (for any OS). SMB is commonly known as the Windows file sharing protocol, but is also implemented in other operating systems like Mac OS X and Linux.
Please Note: By design, you can only move files to and from your home directory. Each of the compute nodes has access to this directory.
For either protocol, your username and password are your PennKey KITE credentials.
Windows File Sharing
The HPC3 has Samba/CIFS file sharing enabled for file system browsing and usage.
On Windows
- Right-Click the Start icon (lower left corner) and select (open) File Explorer
- Right-Click Network or This PC and select Map network drive…
- Drive: your choice, I like ‘H’ (HPC3!)
- Folder: \\hpc3-fs.wharton.upenn.edu\username
- Check the ‘Connect using different credentials’ checkbox
- Optionally check the ‘Reconnect at sign-in’ checkbox (only recommended for on-campus desktop systems)
- Click ‘Finish’ button
- Specify “PennKey@kite.upenn.edu” (without the quotes) as your username, NOT just “PennKey” (or some local username)
On Apple OSX
From Finder, open the Go menu > Connect to Server (or Cmd-K)
Address: smb://hpc3.wharton.upenn.edu/PennKey
On Linux
There are enough flavors of Linux that it’s difficult to document the ‘right way’. Essentially, connect to smb://hpc3.wharton.upenn.edu/PennKey, and if you need assistance just let us know.
SSH-FTP
If you are using a file transfer client (such as SecureCRT, etc) use the following settings:
- File Transfer Protocol (if selectable): SSH-FTP
- Host: hpc3.wharton.upenn.edu
The first time you login, you may receive a message similar to “Host key not found from list of known hosts. Are you sure you want to continue connecting?” Answer yes to make the connection. You should not receive this message on subsequent connections.
Please Note: if you are transferring a file from Windows, you should set the transfer mode to ASCII for program and job script files, otherwise line breaks will not translate correctly. If you find a file with incorrect line breaks from Windows, you can use the dos2unix filename command to fix it.
If you are using the command scp, the format is: scp source-filename target-filename. When referencing a remote file, you must use the full syntax for the file (username@remote-system:/remote-filename). Some examples:
- From your local computer, copy a file from your local computer to the HPC3: scp path/to/local-file username@hpc3.wharton.upenn.edu:path/to/hpc3-file
- From your local computer, copy a file from the HPC3 to your local computer: scp username@hpc3.wharton.upenn.edu:path/to/hpc3-file path/to/local-file
Please Note: if you are transferring files a large amount of files between your local computer and the HPC3, it is much more efficient to tar or zip them into a single file and untarring or unzipping once its transferred. For more details, check out the manual pages for the commands: man tar / man zip / man unzip.