HPCC Dropbox Client Retirement

Dropbox on Wharton’s HPC Cluster has been a great tool for many of you! The ability to sync files between desktop, HPCC, home, phone, and collaborators is of immense value. However, Dropbox has announced that on November 7th, 2018 Dropbox will no longer support our setup* with their official client, making it impossible for us to continue to use their client in the cluster.

That’s a bit of bad news for those of you who have worked hard to build your workflow around a Dropbox client file syncing model. If we had a choice, we would continue to support the vendor’s official client in our environment. Sadly, we do not.

The Good News

But wait! There is good news!

While the Dropbox client has been useful to many, it really wasn’t made for multi-user systems—like our cluster. So over the years it’s also been the source of a lot** of issues for our user community, and a correspondingly-large amount of work for your HPC Team. Even before we heard the news from Dropbox, we were looking at better, more ‘cluster friendly’ ways to work with your Dropbox content across the cluster.

rclone

We were looking for a product that:

  • didn’t need to run as a service (always on), like the Dropbox client. These are extremely ‘expensive’ when used by many users on a multi-user system, and Dropbox was one of the most expensive around.
  • would be able to sync to and from not just Dropbox, but some other vendors, too, particularly:
    • Amazon S3
    • Box
    • Google Drive
    • Microsoft OneDrive
  • would be simple to set up, and easy to implement into cluster workflow
  • could run from anywhere on the cluster, not just the login nodes!

Enter rclone! Spend a few minutes configuring your ‘remotes’ on the cluster via the ‘rclone config’ command (see full instructions here!!), then you’re ready to sync, copy, etc to and from whichever 3rd party cloud storage system you like, all using the same command structure, which is really useful. So I can change my 3rd party cloud storage solution from Dropbox to Box, but still use the same commands to move / sync files, just changing the remote I’m connecting to. Neat!

Further Details or Questions

For further details, please see our rclone Page, or get in touch via research-computing@wharton.upenn.edu. Thank you for your understanding, and we hope rclone provides you with an excellent 3rd party cloud storage syncing experience, as it does us.

 

* particularly: a specific C library version requirement, and a new filesystem type requirement, both of which are impossible to implement in our cluster

** over 130 tickets opened in the last 5 years, one of our recurring pain points

About

With two decades of experience supporting research and more than a decade at The Wharton School, Hugh enjoys the challenges and rewards of working with world-class researchers doing Amazing Things with research computing. Robust and scalable computational solutions (both on premise and in The Cloud), custom research programming solutions (clever ideas, simple code), and holistic, results-focused approaches to projects are the places where Hugh lives these days. On weekends you're likely to find him running through the woods with a topo map and compass, orienteering.