<p>
As a warning before we dive into things, this post is less of a formal publication and more of a stream of conscience.
</p>

<p>
My employer <a href="http://newcars.com/jobs/">newcars.com</a> has allowed the technical staff to host hackathon! Over the past couple weeks I have had quite a few ideas tumbling around in my head:

<ul>
	<li>Standup a central logging server</li>
         My colleague <a href="http://bobbylikeslinux.net/">Bobby</a> ran with this idea, he posted his progress here: https://bitbucket.org/robawt/robawt-salt-logstash/overview
	<li>Standup Sensu for monitoring</li>
        <strong>Update:</strong> I tested setup a test environment with <a href="http://russell.ballestrini.net/sensu-salt">sensu-salt and salt-cloud</a>.
	<li>Bake-off and document some KVM virtualization hypervisors (Ubuntu or SmartOS)</li>
	<li>Test Docker and document findings</li>
</ul>
</p>

<p>
Ultimately I have chosen to dedicate my time to testing out virtualization on the new Cisco UCS Blade servers. I plan to test Ubuntu KVM first. 
</p>

<p>
<strong>KVM (Ubuntu 12.04)</strong>
</p>

<p>
I decided to use a Salt-stack configuration management formulas to document how I transformed <code>kvmtest02</code> (a regular Ubuntu 12.04 server) into a KVM hypervisor.  I use <code>kvmtest0a</code> and <code>kvmtest02b</code> when refering to the virtual machines living on the hypervisor.  Here is the formula:
</p>

<p>
<strong>kvm/init.sls:</strong>
<pre lang="yaml">
# https://help.ubuntu.com/community/KVM
# Ubuntu server a KVM - Kernel Virtual Machine hypervisor

# the official ubuntu document suggests we install the following
kvm-hypervisor:
  pkg.installed:
    - names:
      - qemu-kvm
      - libvirt-bin
      - ubuntu-vm-builder
      - bridge-utils

# we need to make all of our ops people part of the libvirtd group
# so that they may create and manipulate guests. we skip this for now
# and assume all virtual machines will be owned by root.

# these are optional packages which gives us a GUI for the hypervisor
virt-manager-and-viewer:
  pkg.installed:
    - names:
      - virt-manager 
      - virt-viewer 
    - require:
      - pkg: kvm-hypervisor

# create a directory to hold virtual machine image files
kvm-image-dir:
  file.directory:
    - name: /cars/vms
    - user: root
    - group: root
    - mode: 775

# we need this package so we can build a network bridge interface
# so that our virtual machines get a real identity on the network
bridge-utils:
  pkg.installed:
    - name: bridge-utils
</pre>
</p>

<p>
I used the following to install the formula to the test hypervisor:

<pre lang="bash">
salt 'kvmtest02.example.com' state.highstate
</pre>
</p>

<p>
The salt highstate reported everything was good (green), so I moved on to setting up the bridge networking interface.  At this point I don&#8217;t want to figure out the logistics setting up the network bridge in configuration management, so I simply manually edited <code>/etc/network/interfaces</code> to look like this (substitute your own network values):
<pre lang="text">

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#iface eth0 inet manual

# https://help.ubuntu.com/community/KVM/Networking
# create a bridge so guest VMs may have their own identities
auto br0
iface br0 inet static
    address XXX.XX.89.42
    netmask 255.255.255.0
    network XXX.XX.89.0
    broadcast XXX.XX.89.255
    gateway XXX.XX.89.1

    # dns-* options are implemented by the resolvconf package
    dns-nameservers XXX.XX.254.225 XXX.XX.254.225
    dns-search example.com

    # bridge_* options are implemented by bridge-utils package
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0
</pre>

Then, I crossed my fingers and reloaded the network stack using this command:

<pre>
/etc/init.d/networking restart
</pre>

I also used this to &#8220;bounce&#8221; the bridge network interface:

<pre>
ifdown br0 &#038;&#038; ifup br0
</pre>

I verified with <code>ifconfig</code>.

</p>


<p>
I&#8217;m ready to create my first VM.  There are many different ways to boot the VM and install the operating system.  KVM is fully virtualized so nearly any operating system may be install on the VM.
</p>

<p>
If you have not already, please get familiarized with the following two commands: 
<ol>
  <li>virsh</li>
  <li>virt-install</li>
</ol>
</p>

<p>
The <code>virsh</code> command is an unified tool / API for working with hypervisors that support the libvirt library.  Currently <code>virsh</code> supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox and VMware ESX.  For more information run <code>virsh help</code>.
</p>

<p>
The <code>virt-install</code> command line tool is used to create new KVM, Xen, or Linux container guests using the &#8220;libvirt&#8221; hypervisor management library. For more information run <code>man virt-install</code>.
</p>

<blockquote>
Woah, virsh and virt-install both support LXC?
</blockquote>

<p>
We have decided to only support Ubuntu 12.04 at this time, so obviously we will choose that for our guest&#8217;s OS.  Now we need to decide on an installation strategy. We may use the following techniques to perform an install: 
<ul>
  <li>boot from local CD-rom</li>
  <li>boot from local ISO</li>
  <li>boot from PXE server on our local vLAN</li>
  <li>boot from netboot image from anywhere in the world</li>
</ul>

We will choose the PXE boot strategy because our vLAN environment already uses that for physical hosts.
</p>

<p>
We will use the <code>virt-install</code> helper tool to create the virtual machine&#8217;s &#8220;hardware&#8221; with various flags.  Lets document the creation of this guest in a simple bash script so we may reference it again in the future.
</p>

</strong>/tmp/create-kvmtest02-a.sh:</strong>
</p>
<pre lang="bash">
HOSTNAME=kvmtest02-a
DOMAIN=example.com

sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $HOSTNAME \
  --vcpu 2 \
  --ram 4096 \
  --disk /cars/vms/$HOSTNAME.qcow2,size=20 \
  --os-type linux \
  --graphics vnc \
  --network bridge=br0,mac=RANDOM \
  --autostart \
  --pxe
</pre>
</p>

<p>
This was not used, but shows the flags to perform a netboot from Internet:
<pre lang="bash" style="font-size: 0.8em;">
--location=http://archive.ubuntu.com/ubuntu/dists/raring/main/installer-amd64/ \
--extra-args="auto=true priority=critical keymap=us locale=en_US hostname=$HOSTNAME domain=$DOMAIN url=http://192.168.1.22/my-debconf-preseed.txt"
</pre>
</p>

<p>
I created the vm:
<pre lang="bash">
bash /tmp/create-kvmtest02-a.sh
</pre>
</p>

<p>
<code>virt-install</code> drops you into the &#8220;console&#8221; of the VM, but this will not work yet, so we use ctrl+] to break out and get back to our hypervisor.  Use <code>virsh list</code> to list all the currently running VMs.

Lets use <code>virt-viewer</code> to view the VMs display.  For this we need to SSH to the hypervisor and forward our display to our workstation, we do this with the <code>-X</code> flag. For example:

<pre lang="bash">  
ssh -X kvmtest02
</pre>

Now we can launch <code>virt-viewer</code> on the remote hypervisor, and the GUI will be drawn on our local X display!

<pre lang="bash">  
virt-viewer kvmtest02-a
</pre>

Once I got that to work, I also tested <code>virt-manager</code> which gives a GUI to control all guests on the remote hypervisor.

<pre lang="bash">  
virt-manager
</pre>

</p>


 Now we need to determine the auto-generated MAC Address of the new virtual machine.

<pre lang="bash">
virsh dumpxml kvmtest02-a | grep -i "mac "
  mac address='52:54:00:47:86:8e'
</pre>

We need to add this MAC address to our PXE server&#8217;s DHCP configuration to allocate the IP and tell it where to PXE-boot from.
</p>

<p>
During a real deployment we would get an IP address allocated and an A record and PTR setup for new servers.  This is a test and I will be destroying all traces of this virtual machine after presenting during the hackathon, so for now I&#8217;m going to skip the DNS entries and &#8220;steal&#8221; an IP address.  I must be VERY careful not to use an IP address already in production.  First use dig to find an IP without a record, then attempt to ping and use NMAP on the IP.
</p>

<pre lang="bash">
dig -x XXX.XX.89.240 +short
ping XXX.XX.89.240
nmap XXX.XX.89.240 -PN
</pre>

<p>
The IP address checked out, it didn&#8217;t have a PTR, it didn&#8217;t respond to pings, and using nmap proved there were no open ports.  I&#8217;m very confident this IP address is not in use.
</p>

<p>
I added a record to our DHCP / PXE server for this Virtual Machine.  I attempted multiple times to pxe boot the VM, but the network stack was never automatically configured&#8230;  The DHCP server was discovering the new VMs MAC and offering the proper IP address, as noted by these log lines:

<pre lang="bash">
Dec 13 07:57:43 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:43 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
</pre>

I wasted about 4 hours attempting to troubleshoot and diagnose why the VM wouldn&#8217;t work with DHCP.  I ended the night without any guests online&#8230;
</p>

<p>
<strong>The Next DAY!</strong>
</p>

<p>
So today I decided to stop trying to get DHCP and PXE working.  I downloaded an Ubuntu server ISO to the hypervisor, and used <code>virt-manager</code> to mount the ISO on the guest and booted for a manual operating system install.  

<p>
This did two things, it proved that the hypervisor&#8217;s network bridge <code>br0</code> worked for static network assigned settings and that something between the DHCP server and the hypervisor was preventing the <code>DHCPOFFER</code> answer from getting back to the VM.  I looked into iptables firewall, removed apparmor, removed SELINUX and reviewed countless logs looking for hints&#8230; then moved on&#8230;
</p>

<p>
I was able to get salt-minion installed on the vm using our post-install-salt-minion.sh script, which I manually downloaded from the salt master.  But to keep this test self contained, I pointed the VM&#8217;s salt-minion to <code>kvmtest02</code> which we already had setup as a test salt-master.
</p>

<p>
The salt-master saw the salt-minion&#8217;s key right away, so I decided to target an install.  This what I applied to the VM:
</p>

<p>
salt/top.sls:
<pre lang="yaml">
  'kvmtest02.example.com':
    - kvm

  'kvmtest02b.example.com':
    - virtualenv
    - python-ldap
    - nginx
    - the-gateway
</pre>

<p>
salt-pillar/top.sls
</p>
<pre lang="yaml">
  # kvmtest02b gateway in a VM experiment
  'kvmtest02b.example.com':
    - nginx
    - the-gateway.alpha
    - deployment-keys.the-gateway-alpha
</pre>
</p>

<p>
The stack was successfully deployed to the VM and proved that virtual machines are a viable solution for stage or production.  It also gave me the change to test out this particular deployment again and found a few gotchas we need to create maintenance tickets for.  
</p>

<p>
Without configuration management, it would have taken weeks to deploy this custom application stack.  The install with configuration management took less then 10 minutes!  
</p>

<p>
One of the KVM related snags I ran into was that Nginx does some fun calculations with cpu cache to determine hash table sizes.  As a temporary work around, until I can devote more research time, I raised up the following three hash table directives in the http section of </code>nginx.conf</code>:
</p>

<pre>
    server_names_hash_bucket_size 512;
    types_hash_bucket_size 512;
    types_hash_max_size 4096;
</pre>

</p>

<p>
<strong>SmartOS</strong>
</p>

<p>
<strong>snippet from /etc/dhcp/dhcpd.conf</strong>
<pre>
# SmartOS hypervisor group to boot image
group "smartos-hypervisors" {
  next-server xxx.xx.89.71;

   host smrtest01-eth0 {
        hardware ethernet 00:25:B5:02:07:DF;
        option host-name "ncstest01";
        fixed-address smrtest01.example.com;

        if exists user-class and option user-class = "iPXE" {
           filename = "smartos/menu.ipxe";
        } else {
           filename = "smartos/undionly.kpxe";
        }
    }

}

</pre>

<pre>
mkdir /cars/tftp/smartos
cd /cars/tftp/smartos
wget http://boot.ipxe.org/undionly.kpxe
wget https://download.joyent.com/pub/iso/platform-latest.tgz
tar -xzvf platform-latest.tgz
mv platform-20130629T040542Z 20130629T040542Z
mkdir platform
mv i86pc/ platform/
</pre>

create boot menu that we referenced, <code>vim /cars/tftp/smartos/menu.ipxe</code>

<pre>
#!ipxe

kernel /smartos/20130629T040542Z/platform/i86pc/kernel/amd64/unix
initrd /smartos/20130629T040542Z/platform/i86pc/amd64/boot_archive
boot
</pre> 

Make sure to replace platform version with current.

</p>

<p>
I was able to get the blade server to PXE boot the image, but it seems SmartOS doesn't really support the SANs.  SmartOS really expects to see local disks, and to build a ZFS pool on top of that.  Basically SmartOS could be used to build a SAN, so they didn't put much effort in supporting SANs.  After I figured this out I abandoned this test.  We could revist this again, using one of the Dell servers, or use it to stand up a really powerful Alpha server environment.
</p>

<p>
<strong>LXC</strong>
</p>
 
<p>
Run /usr/bin/httpd in a linux container guest (LXC). Resource usage is capped at 512 MB of ram and 2 host cpus:

<pre lang="bash">
virt-install \
--connect lxc:/// \
--name lxctest02-a \
--ram 512 \
--vcpus 2 \
--init /usr/bin/httpd
</pre>
</p>
 

<p>
<strong>Discussion points</strong>
</p>
<p>
<ul>
<li>
Why doesn't DHCP work on bridge?
</li>
<li>
If we use virtualization, we need to come up with a plan for IP addresses, like possibly allocate ~5 IP addresses to a hypervisor host
</li>
<li>
We need to come up with a naming convention for guests, in testing I appended a letter to the hypervisor name <code>kvmtest02</code> so the guests names were <code>kvmtest02a</code> and <code>kvmtest02b</code>, is this plausible going forward?
</li>
</ul>
</p>

<p>
<strong>If I had more time ...</strong>
</p>
<p>
<ul>
<li>
I would have liked to test out LXC
</li>
<li>
I would have liked to test out Docker
</li>
<li>
I would have liked to test out physical to virtual migrations
</li>
</p>