Compcache on Ubuntu on Amazon EC2

Posted on May 4, 2010 by heuristicandrew in R bloggers | 0 Comments

[This article was first published on Heuristic Andrew, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The following fully-automatic Bash script downloads, compiles, and initializes compcache version 0.6.2 on Ubuntu Karmic Koala (9.10). This script creates two swaps with a maximum of 4GB uncompressed size each. Two swaps are used to take advantage of 2 CPUs (or CPU cores in a multicore CPU).

Compcache is a fascinating memory compression system. The old alternative to compcache is to use a swap file backed by a hard disk (without using compression), but this kind of swapping is extremely slowed to physical memory or memory compressed by compcache. Compcache creates a swap device backed by compressed memory or by another swap file. Basically, you get more RAM—almost for free—though compcache adds a small overhead for compression. Compression should actually increase performance when backed by a hard drive because it reduces expensive disk I/O. When memory pages are empty (which happens more often than you’d think), compcache stores nothing. I run data analysis using party in R, and it needs huge amounts of memory (well over 10GB). Amazon EC2 gives me a large starting memory capacity, and compcache extends the physical RAM.

The script doesn’t install or make permanent changes, so if you don’t like it, reboot to start over. It was tested on Amazon EC2 with Canonical 64-bit server (ami-55739e3c), and it should work fine with other Ubuntu versions such as the new Lucid Lynx LTS (10.04). For other ways to use Compcache, see compcache: CompilingAndUsingNew .

DIR=/tmp/compcache-0.6.2
CONTROL=$DIR/sub-projects/rzscontrol/rzscontrol

# install kernel headers
sudo apt-get install linux-headers-$(uname -r) gawk

# show memory before
free
cat /proc/swaps

# compile
cd /tmp
wget http://compcache.googlecode.com/files/compcache-0.6.2.tar.gz
rm -rf $DIR
tar xvzf compcache-0.6.2.tar.gz
cd $DIR
make

# load dependency modules
sudo modprobe lzo_compress
sudo modprobe lzo_decompress

# load ramzswap module
sudo insmod ramzswap.ko num_devices=2
sleep 1s

# Initialize devices with 4GB each upper limit of uncompressed memory.  This device does not have a backing swap.
sudo $CONTROL /dev/ramzswap0 --disksize_kb=4456448 --init
sudo $CONTROL /dev/ramzswap1 --disksize_kb=4456448 --init

# activate
sudo swapon -p 5 /dev/ramzswap0
sudo swapon -p 5 /dev/ramzswap1

# show stats
free
cat /proc/swaps
sudo $CONTROL /dev/ramzswap0 --stats