Compcache on Ubuntu on Amazon EC2

May 4, 2010

(This article was first published on Heuristic Andrew, and kindly contributed to R-bloggers)

The following fully-automatic Bash script downloads, compiles, and initializes compcache version 0.6.2 on Ubuntu Karmic Koala (9.10). This script creates two swaps with a maximum of 4GB uncompressed size each. Two swaps are used to take advantage of 2 CPUs (or CPU cores in a multicore CPU).

Compcache is a fascinating memory compression system. The old alternative to compcache is to use a swap file backed by a hard disk (without using compression), but this kind of swapping is extremely slowed to physical memory or memory compressed by compcache. Compcache creates a swap device backed by compressed memory or by another swap file. Basically, you get more RAM—almost for free—though compcache adds a small overhead for compression. Compression should actually increase performance when backed by a hard drive because it reduces expensive disk I/O. When memory pages are empty (which happens more often than you’d think), compcache stores nothing. I run data analysis using party in R, and it needs huge amounts of memory (well over 10GB). Amazon EC2 gives me a large starting memory capacity, and compcache extends the physical RAM.

The script doesn’t install or make permanent changes, so if you don’t like it, reboot to start over. It was tested on Amazon EC2 with Canonical 64-bit server (ami-55739e3c), and it should work fine with other Ubuntu versions such as the new Lucid Lynx LTS (10.04). For other ways to use Compcache, see compcache: CompilingAndUsingNew .


# install kernel headers
sudo apt-get install linux-headers-$(uname -r) gawk

# show memory before
cat /proc/swaps

# compile
cd /tmp
rm -rf $DIR
tar xvzf compcache-0.6.2.tar.gz
cd $DIR

# load dependency modules
sudo modprobe lzo_compress
sudo modprobe lzo_decompress

# load ramzswap module
sudo insmod ramzswap.ko num_devices=2
sleep 1s

# Initialize devices with 4GB each upper limit of uncompressed memory.  This device does not have a backing swap.
sudo $CONTROL /dev/ramzswap0 --disksize_kb=4456448 --init
sudo $CONTROL /dev/ramzswap1 --disksize_kb=4456448 --init

# activate
sudo swapon -p 5 /dev/ramzswap0
sudo swapon -p 5 /dev/ramzswap1

# show stats
cat /proc/swaps
sudo $CONTROL /dev/ramzswap0 --stats

  • To leave a comment for the author, please follow the link and comment on their blog: Heuristic Andrew. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

    If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

    Tags: , ,

  • Comments are closed.


    Never miss an update!
    Subscribe to R-bloggers to receive
    e-mails with the latest R posts.
    (You will not see this message again.)

    Click here to close (This popup will not appear again)