MCMC in The Cloud

Arun Gopalakrishnan, a doctoral candidate in Wharton’s Marketing department, recently approached me to discuss taking his MCMC simulations in R to the next level: Big. His paper is under review at a journal, and the referees asked for more. Much more.

AWS LogoWhile he sharpened his code, Wharton Computing’s Research Computing team built, and tested a 16-node, 512-thread computing cluster in the Amazon Web Services EC2 environment.

Instead of taking weeks (or even months) to build it ourselves, followed by potentially years of sub-optimal use (while consuming power, rack, and cooling resources), with the use of StarCluster and AWS we created Arun’s custom, personal cluster in about an hour. He can launch it in about 10 minutes any time he wishes, and shut it down when not in use.

It costs pennies per day when not in use, and $0.34/hour (currently) per node per hour when in use. He can scale it from 32-threads to 1024-threads either at startup, or on the fly.

Thanks to the StarCluster crew’s great free product, and AWS’s powerful, flexible, inexpensive services, Arun was able to go from single-thread jobs on his laptop to a 480-thread job in The Cloud in only a few short days.

About

With two decades of experience supporting research and more than a decade at The Wharton School, Hugh enjoys the challenges and rewards of working with world-class researchers doing Amazing Things with research computing. Robust and scalable computational solutions (both on premise and in The Cloud), custom research programming solutions (clever ideas, simple code), and holistic, results-focused approaches to projects are the places where Hugh lives these days. On weekends you're likely to find him running through the woods with a topo map and compass, orienteering.