IO Benchmarking Part 1 - Physical and RAID

Please note: This article is still a work in progress. The experiment is still ongoing.
The goal of this mini-project is to answer a question that has been bugging me for a while:

Which, of the multitude of options available, makes the best configuration for a Linux KVM host's storage?

In part one I hope to work out which performs better on the block level, below the filesystem.

Hardware

The purpose here is to compare configurations against each other, not to compare against some external baseline and certainly not to set some sort of speed record.

I bought a second hand Dell 2950 server from ebay. It has 16Gb of RAM, a pair of Xeon CPUs. It's a few years old but in good condition and seems rock solid stable.

The server has a <RAID Controller details here>

I found four of the same model 1Tb hard drives. These are Western Digital Blue drives which are again not going to blow anyone's mind with performance.

I have one 120Gb Intel 330 SSD which will be used for the various SSD caching setups. More on that in a moment.

Software

Operating System

  • Ubuntu Server 12.04 Precise amd64 with latest updates as of 21 July 2013 <insert dpkg listing>
  • Mainline kernel 3.10.1 <more detail>

I will be using Ubuntu 12.04 as the base OS with the latest (at time of writing obviously) main line kernel.

I'm using the latest main line kernel so I can try out some of it's new caching features: bcache, EnhanceIO and dm-cache. Each of these offer their own implementation of using SSDs as a cache for slow mechanical disks. My initial impression from what I have read about these is that the really do a very good of improving IO performance but compared with each other the difference is minimal. We will see.

Phoronix Test Suite setup

phoronix-test-suite openbenchmarking-login "shaunmaher"
phoronix-test-suite batch-setup
These are the default configuration options for when running the Phoronix Test Suite in a batch mode (i.e. running phoronix-test-suite batch-benchmark universe). Running in a batch mode is designed to be as autonomous as possible, except for where you'd like any end-user interaction.

    Save test results when in batch mode (Y/n): y
    Open the web browser automatically when in batch mode (y/N): n
    Auto upload the results to OpenBenchmarking.org (Y/n): y
    Prompt for test identifier (Y/n): n
    Prompt for test description (Y/n): n
    Prompt for saved results file-name (Y/n): n
    Run all test options (Y/n): n

Now we run the “pts/server” suite to get a baseline for the server itself. It looks like this.

We have four disks to play with so we have a few RAID options. The priorities are in this order:

  • Fault tolerance; then
  • Performance; then
  • Space

Performance above space means RAID 10. We might come back later and throw some RAID 5 or RAID 6 into the mix but for now RAID 10 is the go.

Each competitor will host an LVM volume group with one or more Logical Volumes. This will become important later when we're testing different Virtual Disk storage mechanisms.

This is what we want to compare:

  • A single disk - so we have a baseline to compare with.
  • A single SSD - again, so we know what we're working with
  • The Dell's PERC RAID controller - This will be our token “Hardware RAID” solution.
  • Linux LVM RAID 10
  • Single disk + bcache
  • PERC RAID controller + bcache
  • Linux MD RAID 10 + bcache
  • Linux LVM RAID 10 + bcache
  • Single disk + EnhanceIO
  • PERC RAID controller + EnhanceIO
  • Linux MD RAID 10 + EnhanceIO
  • Linux LVM RAID 10 + EnhanceIO
  • Single disk + dm-cache
  • PERC RAID controller + dm-cache
  • Linux MD RAID 10 + dm-cache
  • Linux LVM RAID 10 + dm-cache

Ultimately we want to know which competitor will be the best for hosting the virtual disks for KVM VMs but we have a few layers between our block level storage and the virtual disks that are outside the scope of part of our experiment. For now, a more synthetic benchmark will need to do.

DD

The most simple test we can do, write a bunch of zeros to disk. The results aren't really that useful because writing a bunch of sequential zeros to storage isn't something that happens very often in real world usage but it might at least give us a broad idea as to how each competitor is doing as well as alert us to any red flags that suggest we have stuff set up wrong.

Phoronix Test Suite

The Phoronix Test Suite seems pretty through. I'll let it speak for itself.

Phoronix Test Suite inside a Windows 7 VM

This is going to be our stand in for a Windows Server that does some Windows Servery type things. To keep things fair we will be using an LVM volume as a virtual disk to make sure we don't have any filesystem optimisations, weaknesses, caching, etc. skewing our results.

The VM has a fresh install of Windows 7 with all the latest Windows Updates installed. It is using virtio for it's virtual disks/controllers and has the latest virtio drivers installed.

A disk image with everything setup ready to run the benchmark will be copied into the LVM volume before each test to ensure the previous tests have no effect of future tests, such as fragmenting the guest file system.

Random note

mklabel gpt
mkpart primary ext4 1M 100Gb
set 1 lvm on

sudo pvcreate /dev/sdb1
sudo vgcreate vg_BENCHMARK /dev/sdb1
sudo lvcreate -l 100%FREE -n lv_BENCHMARK vg_BENCHMARK