Using Packer to Build Images for the Acropolis Hypervisor

Packer is a tool from Hashicorp, responsible for Vagrant, Terraform and Vault among others, that I have had a lot of experience on the vSphere and AWS side but hardly any on the KVM / OpenStack side until joining Nutanix. Like most tooling around the “Infrastructure-as-Code” mindset, internally I categorize these rather as “Infrastructure-as-Text-Files” because I can define the build of a VM in a text file, json to be specific, instead of interactively installing an OS. Just in general, I really appreciate this technique as I can use a Vagrantfile to define a self-contained lab, or I can use a Dockerfile to describe a container for example.

Packer actually can simultaneously generate images across multiple platforms as I’ve used it to build multiple images across my vSphere and AWS projects. One would think that for having OVF’s and OVA’s that we would at least have some measure of easy portability across environments, but alas here we are.  In AWS, you have your AMIs and in vSphere, you have an entirely separate vmdk/OVA/OVF and of course there are good reasons for both as each has their own constituent components. 

Since the Acropolis hypervisor is built on a fork of the open source KVM hypervisor, we can use QEMU and Packer to facilitate building images on it, but there is a small learning curve if you are new to Packer or coming from a vSphere environment.

On your workstation of choice, Packer works with Virtualbox, Fusion, Workstation, or Parallels, you will need a linux VM with QEMU installed with your OS and the VT instruction set exposed to the VM. Here is an example with Virtualbox:

And with VMware Fusion:

I would recommend installing CentOS 6.5+ or Ubuntu 14.04+ as a guest VM to create your Packer workstation. Make sure to install the "Virtualization Host" option or install the qemu-kvm packages after you do your base install. Some examples of what this may look like:

Next download and install directions for Packer are here. Once you have it installed, you need a json file specific to the VM you want to build to run with it:
packer build example_os.json

For a starter example of a CentOS VM, you can go to my Github repo here. Clone with: git clone https://github.com/nelsonad77/acropolis-packer-examples.git
In order to see some useful logs, especially if you're building a new or unfamiliar image, I usually run this:
export PACKER_LOG="yes" && export PACKER_LOG_PATH="packer.log" && packer build centos.json

This will do a base installation of CentOS 6.7 and you can customize either the json or the kickstart file as you wish. Some notable specs in the json file are:

"type": "qemu" –Of course this will have to be set as we are building using QEMU

"format": "raw" –Options are RAW or QCOW2. Since the Nutanix file system already handles copy-on-write quite well and has other acceleration already built-in, we just use raw.

"headless": true –This is because I am just running an Ubuntu server from command-line. You have to do this if you're not using a desktop gui to show windows, but since this is deliberately an automated install, the gui is being driven by Packer and not by you anyway.

"accelerator": "kvm" –You can use this if you are building with QEMU installed on your linux virtualization host or you could also use “none”.

"iso_url": "http://mirrors.rit.edu/centos/6.7/isos/x86_64/CentOS-6.7-x86_64-minimal.iso",

"iso_checksum": "9381a24b8bee2fed0c26896141a64b69" –Feel free to update this to your ISO of choice and make sure that the checksum matches or of course the build will not commence successfully.

"disk_size": 5000 –Default initial disk size, feel free to customize.

"output_directory": "output_image"

"vm_name": "centos67template.img" –Assuming the build completes successfully, this is the .img file and path to look in that you can import with the Prism Image Configuration service.

"provisioners":

  [

    {

      "type": "shell",

      "inline": [

        "sleep 3",

        "echo \"NOZEROCONF=yes\" >> /etc/sysconfig/network",

        "adduser nutanix-admin",

        "echo 'nutanix-admin:nutanix' |chpasswd",

        "mkdir /home/nutanix-admin/.ssh",

        "chown nutanix-admin:nutanix-admin /home/nutanix-admin/.ssh",

        "chmod 700 /home/nutanix-admin/.ssh",

        "echo \"nutanix-admin ALL=(ALL) ALL\" >> /etc/sudoers"

      ]

    },

    {

      "type": "file",

      "source": "centos.json",

      "destination": "/root/centos.json",

      "source": "httpdir/centos6-ks.cfg",

      "destination": "/root/centos6-ks.cfg"

    },

Under provisioners you can see some examples of things you can do to your image. You have the ability to add users, copy in ssh keys, install and update packages and otherwise prep the image baseline exactly as you want. However, I would also caution against adding too much in the default image as individual app-specific packages can be added with your Chef, Ansible, or configuration management tool of choice after you clone your images.

Once the image build completes, you can login to your Nutanix cluster and upload your .img file with the Image Configuration service as a DISK instead of an ISO. From there, you can create a new template VM, choose to clone the disk from the Image service and power it on to verify that you have a new working template that you didn't even have to interactively install.

From there you can prep the image for mass cloning as I described here, if you didn't already perform these steps with the Packer inline provisioning, and then immediately move on to deploying and scaling out your apps.

There are not as many KVM examples in the wild as there are for AWS or vSphere, which is one of the main reasons I wrote this post, but at least there are other builds that can be adapted like the Bento boxes from Chef here: https://github.com/chef/bento

If you find this useful, I would encourage you to share out your own public packer build files if possible.

Additional links:

https://www.packer.io/docs/builders/qemu.html

Acropolis Image and Cloning Primer for Automation

Part of scaling out any environment is having a good template that can be spun up and incorporated easily into your imaging and configuration management prep before it’s added to your platform of choice. For Nutanix, I wanted to get these base templates ready and share what I needed to do for Ubuntu/Debian, CentOS/RHEL, and CoreOS:

Ubuntu 14.04

1. Run any updates and add any packages you want to be included in your baseline template with sudo apt-get. Keep in mind that for most packages and configurations, you will want as minimal an image as possible and add those components with Chef, Ansible, etc. Shared keys may be copied to an admin user's ~/.ssh directory to prepare for passwordless SSH.

2. Precreate DNS entries for VMs so that when clones boot they pull hostname from DHCP.

3. Add a ‘hostname’ script under /etc/dhcp/dhclient-exit-hooks.d:

#!/bin/sh

# Filename:     /etc/dhcp/dhclient-exit-hooks.d/hostname

# Purpose:      Used by dhclient-script to set the hostname of the system

#               to match the DNS information for the host as provided by

#               DHCP.

#

# Do not update hostname for virtual machine IP assignments

if [ "$interface" != "eth0" ] && [ "$interface" != "wlan0" ]

then

    return

fi

if [ "$reason" != BOUND ] && [ "$reason" != RENEW ] \

   && [ "$reason" != REBIND ] && [ "$reason" != REBOOT ]

then

        return

fi

echo dhclient-exit-hooks.d/hostname: Dynamic IP address = $new_ip_address

hostname=$(host $new_ip_address | cut -d ' ' -f 5 | sed -r 's/((.*)[^\.])\.?/\1/g' )

echo $hostname > /etc/hostname

hostname $hostname

echo dhclient-exit-hooks.d/hostname: Dynamic Hostname = $hostname

4. Make the ‘hostname’ script ready: chmod a+r hostname

5. Poweroff the VM that will become the template.

CentOS 6.5

1. Run any updates and add any packages you want to be included in your baseline template with yum or the appropriate package manager. Keep in mind that for most packages and configurations, you will want as minimal an image as possible and add those components with Chef, Ansible, etc. Shared keys may be copied to an admin user's ~/.ssh directory to prepare for passwordless SSH.

2. Precreate DNS entries for VMs so that when clones boot they pull their hostname from DHCP.
3. Remove or delete the HWADDR line from the /etc/sysconfig/network-scripts/ifcfg-eth0 file.

4. Remove the mapped network device so that when a new clone boots, it grabs a new MAC and is able to re-use eth0: rm /etc/udev/rules.d/70-persistent-net.rules

5. You can leave the hostname at localhost.localdomain since the new hostname will be mapped on boot up from the DNS record lookup.

6. Poweroff the VM that will become the template.

CoreOS

CoreOS is a special case as it deliberately is “Just-enough-OS” to run containers. To get into the really interesting work of CoreOS with Kubernetes and Tectonic, you’ll need a cluster master(s) with more details around cluster architecture from CoreOS: https://coreos.com/os/docs/latest/cluster-architectures.html
For cloning mass amounts of nodes, you’ll want to create your own cluster member/worker/minion template that feeds into that master with a cloud-config file on a config-drive. Configuring an etcdmaster is here until I customize my own procedure. 

1. Download the latest ISO from the stable, beta, or alpha release channel. You can also set this when you install to disk.

2. Make sure you have one or more ssh key(s) generated on the host/desktop/laptop you would like to use to connect to any of these worker nodes for individual configuration: ssh-keygen –t rsa –b 2048. Also, since you'll basically have a shared ssh key across the hosts at this point, Ansible can take advantage of this easily, more info from CoreOS here and from me here.

3. Create a user_data text file according to directions here.

4. Create a user_data text file with your favorite text editor. You’ll need a few changes for an etcdmaster node, but here is an example cloud-config file from coreos here:

#cloud-config

ssh_authorized_keys:

-ssh-rsa <copy the entirety of your id_rsa.pub file here>

coreos:

            etcd2:

                        proxy: on

                        initial-cluster: etcdserver=http://<etcd-master-ip-here>:2380

                        listen-client-urls: http://localhost:2379

             fleet:

                        etcd_servers: http://localhost:2379

                        metadata: "role=etcd"

            units:

                        - name: etcd2.service

                        command: start

                        - name: fleet.service

                        command: start

5. Convert that text file into an iso according to the directions copied from here. Depending on your OS you may use mkisofs instead of hdiutil: 

mkdir –p /tmp/new-drive/openstack/latest

cp user_data /tmp/new-drive/openstack/latest

hdiutil makehybrid -iso -joliet -default-volume-name config-2 -o configdrive.iso /tmp/new-drive                                                             

rm –r /tmp/new-drive

6. Upload both the config-drive ISO and your chosen CoreOS ISO to the Nutanix image service.

7. Create your base CoreOS VM and attach both ISO’s and boot the VM.

8. When your VM boots, it should auto-login as the ‘core’ user and you can run the install to disk: sudo coreos-install –d /dev/sda –C stable –c /media/configdrive/openstack/latest/user_data

9. Eject both ISO’s and power off the template: sudo poweroff.

Mass Cloning in acli

Using either the Acropolis CLI or REST API, you can set up the quick provisioning of a massive number of clones, given the number of nodes at your disposal.

1. To clone the VMs using the Acropolis CLI, ssh into either the cluster IP or a CVM IP as user Nutanix.

2. At the command line, enter ‘acli’ and feel free to customize the upper bound of VMs that you can build on your Nutanix cluster (code modified from here https://vstorage.wordpress.com/2015/06/29/bulk-creating-vms-using-nutanix-acropolis-acli/):

for n in {1..10}

do

vm.clone vm$n clone_from_vm=basetemplate_name

vm.on vm$n

done

3. If you want to script the operation, just preface the vm.operation with acli, as in:

for n in {1..10}

do

acli vm.clone vm$n clone_from_vm=basetemplate_name

acli vm.on vm$n

done

4. REST examples of cloning VMs via Acropolis on Github, https://github.com/nelsonad77/acropolis-api-examples courtesy of Manish Lohani @ Nutanix.

5. The number of cloned VMs should power on, claim their IP/hostname, and be ready for deploying a configuration with your favorite CM tool.

Additional Links:

http://askubuntu.com/questions/104918/how-to-get-the-hostname-from-a-dhcp-server

https://coreos.com/os/docs/latest/cluster-architectures.html

https://coreos.com/os/docs/latest/config-drive.html

https://coreos.com/os/docs/latest/network-config-with-networkd.html