Arun Gopalakrishnan, a doctoral candidate in Wharton’s Marketing department, recently approached me to discuss taking his MCMC simulations in R to the next level: Big. His paper is under review at a journal, and the referees asked for more. Much more.
While he sharpened his code, Wharton Computing’s Research Computing team built, and tested a 16-node, 512-thread computing cluster in the Amazon Web Services EC2 environment.
Instead of taking weeks (or even months) to build it ourselves, followed by potentially years of sub-optimal use (while consuming power, rack, and cooling resources), with the use of StarCluster and AWS we created Arun’s custom, personal cluster in about an hour. He can launch it in about 10 minutes any time he wishes, and shut it down when not in use.
It costs pennies per day when not in use, and $0.34/hour (currently) per node per hour when in use. He can scale it from 32-threads to 1024-threads either at startup, or on the fly.
Thanks to the StarCluster crew’s great free product, and AWS’s powerful, flexible, inexpensive services, Arun was able to go from single-thread jobs on his laptop to a 480-thread job in The Cloud in only a few short days.