Rclone

rclone is a command line program to copy files and directories to and from a large number of cloud storage solutions, including Box & Dropbox, S3, OneDrive, Google Drive, and many others. It is a straightforward and solid, yet powerful tool for use in our clustered environment, providing copy up and down service to and from our your favorite storage solution.

rclone is installed across PARCC Betty systems, and can be run from any compute server, in an srun --pty bash session or an Open OnDemand (OOD) Interactive Desktop session.

Configuration

Before you begin copying files, you will need to configure rclone for your “remote” (or remotes), the storage systems you wish to connect to. It’s easy! You will need to do this config step in and Open OnDemand (OOD) Desktop session, so that when rclone wants to open a web browser, there is one for it to use. Step-by-step:

  1. open a local browser and start a PARCC Betty Open OnDemand (OOD) session. Choose the “genoa-std-mem” Partition, and “Number of CPUs” to 1. For further details on the Open OnDemand (OOD) service, see PARCC’s Portal Walkthrough
  2. once you have a new desktop running, open a command line terminal by right-clicking the desktop, and ‘Open Terminal Here’, or clicking the Terminal icon in the bottom center menu bar
  3. if you do not yet have the Wharton Software environment set up (source /vast/project/wharton/software/bin/wharton_parcc.sh in your ~/.bashrc file), type source /vast/project/wharton/software/bin/wharton_parcc.sh
  4. if you do not yet have a default Web Browser, you can do that by typing firefox <ENTER> … on the first page you should be asked whether you would like to set Firefox as your default browser. Do that!
  5. start the rclone configuration process by typing: rclone config <ENTER>  … PRO TIP!: rclone config create Dropbox dropbox to skip the below dialog entirely!
$ rclone config
2018/05/15 14:48:06 NOTICE: Config file "/path/to/your/home/directory/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> Dropox
Type of storage to configure.
Choose a number from below, or type in your own value
 7 / Dropbox
   \ "dropbox"
...
Storage> dropbox
Box App Client Id - leave blank normally.
client_id>
Box App Client Secret - leave blank normally.
client_secret>
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...

A browser will open up, and connect to Dropox (or Box, if you chose Box, etc). Authenticate, and then you can ‘Authorize’ Rclone to access your Box / Dropbox files.

That’s it for setup!

Reauthorization

If your token(s) expire(s), you can:

  • Log on to the HPC3 Desktop and open a Mate session
  • Open a Terminal (right click the desktop and “Open in Terminal …”)
  • Type in the reauthorization command: rclone config reconnect Dropbox: (or Box:, etc)
  • Defaults are fine
  • A web browser should pop up, where you can authorize rclone to use your cloud storage service

Syncing Files

While I say ‘syncing‘ and ‘sync‘, please use the ‘copy‘ command for best results!! The best documentation on usage is rclone’s own documentation.

I recommend that you have a single folder within your Box / Dropbox dedicated to your HPCC research files. For example, I have a clusters/betty folder in my Dropbox account. On Betty, I want to locate it at ~/Dropbox/betty, so I copy it down from Dropbox:

$ rclone copy Dropbox:clusters/betty ~/Dropbox/betty -u -v
2026/10/16 07:27:07 INFO : projects/projectA/data/datafile1.parquet: Copied (new)
2026/10/16 07:27:07 INFO : projects/projectA/code/mycoolcode.py: Copied (new)
2026/10/16 07:27:07 INFO : projects/projectA/logs/cooljob-10101011.log: Copied (new)
2026/10/16 07:27:07 INFO : projects/projectA/logs: Set directory modification time (using DirSetModTime)
2026/10/16 07:27:07 INFO : projects/projectA/data: Set directory modification time (using DirSetModTime)
2026/10/16 07:27:07 INFO : projects/projectA/code: Set directory modification time (using DirSetModTime)
2026/10/16 07:27:07 INFO : projects/projectA: Set directory modification time (using DirSetModTime)
2026/10/16 07:27:07 INFO : projects: Set directory modification time (using DirSetModTime)
2026/10/16 07:27:07 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 0 / 0, -, Listed 8
Transferred: 3 / 3, 100%
Elapsed time: 1.3s

2026/10/16 07:27:07 INFO : Dropbox root 'clusters/betty': Committing uploads - please wait...

# examine results
$ find Dropbox/betty/
Dropbox/betty/
Dropbox/betty/projects
Dropbox/betty/projects/projectA
Dropbox/betty/projects/projectA/logs
Dropbox/betty/projects/projectA/logs/cooljob-10101011.log
Dropbox/betty/projects/projectA/code
Dropbox/betty/projects/projectA/code/mycoolcode.py
Dropbox/betty/projects/projectA/data
Dropbox/betty/projects/projectA/data/datafile1.parquet

OPTIONALLY, I have created command line ‘aliases’ to assist with this process. In my ~/.bashrc file, I put:

alias dbdn='rclone copy Dropbox:clusters/betty ~/Dropox/betty -u -v'
alias dbup='rclone copy ~/Dropbox/betty Dropbox:clusters/betty -u -v'
shopt -s expand_aliases   # <- this is so that your aliases are expanded in job scripts

I logged out and back in, and now to copy my ~/Dropbox/betty directory up to the betty directory in my Dropbox account, I just type 'dbup', or 'dbdn' to copy in the other direction. And when I run a job script, I can just add 'dbup' on the line after I’ve done the work and written the output, and Rclone will copy my files to my Dropbox account!

For example:

#!/bin/bash
#SBATCH -J myjob
python mycoolcode.py
dbup

TIP: if you’re running in interactive mode, most software products have a way to run a system or OS command. Take Stata for example, which uses ‘shell’ or ‘!’, like so:

. ! rclone copy ~/Dropbox/betty/projectA/data Dropbox:clusters/betty/projectA/data -u -v
...

# use some data in Stata
. sysuse auto.dta
(1978 Automobile Data)

# save the data to Box directory
. save ~/Dropbox/betty/projectA/data/auto.dta
file ~/Dropbox/betty/projectA/data/auto.dta saved

# copy the directory back up to Box
. ! rclone copy ~/Dropbox/betty/projectA/data Dropbox:clusters/betty/projectA/data -u -v

Unfortunately, aliases aren’t ‘active’ in Stata ‘shell’, I don’t know why (let me know if you do!). You could write a DO files, like:

program dbdn
    ! rclone copy ~/Dropbox/betty/projectA/data Dropbox:clusters/betty/projectA/data -u -v
    end
program dbup
    ! rclone copy Dropbox:clusters/betty/projectA/data ~/Dropbox/betty/projectA/data -u -v
    end

Stash it somewhere (like '~/Dropbox/betty/projectA/rclone_data.do'), and call it from Stata like:

. run ~/ado/rclone_data.do
. dbup
# or
. dbdn

For more advanced use, I recommend looking through the vendor’s documentation.