Using Packer to Build Images for the Acropolis Hypervisor

Packer is a tool from Hashicorp, responsible for Vagrant, Terraform and Vault among others, that I have had a lot of experience on the vSphere and AWS side but hardly any on the KVM / OpenStack side until joining Nutanix. Like most tooling around the “Infrastructure-as-Code” mindset, internally I categorize these rather as “Infrastructure-as-Text-Files” because I can define the build of a VM in a text file, json to be specific, instead of interactively installing an OS. Just in general, I really appreciate this technique as I can use a Vagrantfile to define a self-contained lab, or I can use a Dockerfile to describe a container for example.

Packer actually can simultaneously generate images across multiple platforms as I’ve used it to build multiple images across my vSphere and AWS projects. One would think that for having OVF’s and OVA’s that we would at least have some measure of easy portability across environments, but alas here we are.  In AWS, you have your AMIs and in vSphere, you have an entirely separate vmdk/OVA/OVF and of course there are good reasons for both as each has their own constituent components. 

Since the Acropolis hypervisor is built on a fork of the open source KVM hypervisor, we can use QEMU and Packer to facilitate building images on it, but there is a small learning curve if you are new to Packer or coming from a vSphere environment.

On your workstation of choice, Packer works with Virtualbox, Fusion, Workstation, or Parallels, you will need a linux VM with QEMU installed with your OS and the VT instruction set exposed to the VM. Here is an example with Virtualbox:

And with VMware Fusion:

I would recommend installing CentOS 6.5+ or Ubuntu 14.04+ as a guest VM to create your Packer workstation. Make sure to install the "Virtualization Host" option or install the qemu-kvm packages after you do your base install. Some examples of what this may look like:

Next download and install directions for Packer are here. Once you have it installed, you need a json file specific to the VM you want to build to run with it:
packer build example_os.json

For a starter example of a CentOS VM, you can go to my Github repo here. Clone with: git clone https://github.com/nelsonad77/acropolis-packer-examples.git
In order to see some useful logs, especially if you're building a new or unfamiliar image, I usually run this:
export PACKER_LOG="yes" && export PACKER_LOG_PATH="packer.log" && packer build centos.json

This will do a base installation of CentOS 6.7 and you can customize either the json or the kickstart file as you wish. Some notable specs in the json file are:

"type": "qemu" –Of course this will have to be set as we are building using QEMU

"format": "raw" –Options are RAW or QCOW2. Since the Nutanix file system already handles copy-on-write quite well and has other acceleration already built-in, we just use raw.

"headless": true –This is because I am just running an Ubuntu server from command-line. You have to do this if you're not using a desktop gui to show windows, but since this is deliberately an automated install, the gui is being driven by Packer and not by you anyway.

"accelerator": "kvm" –You can use this if you are building with QEMU installed on your linux virtualization host or you could also use “none”.

"iso_url": "http://mirrors.rit.edu/centos/6.7/isos/x86_64/CentOS-6.7-x86_64-minimal.iso",

"iso_checksum": "9381a24b8bee2fed0c26896141a64b69" –Feel free to update this to your ISO of choice and make sure that the checksum matches or of course the build will not commence successfully.

"disk_size": 5000 –Default initial disk size, feel free to customize.

"output_directory": "output_image"

"vm_name": "centos67template.img" –Assuming the build completes successfully, this is the .img file and path to look in that you can import with the Prism Image Configuration service.

"provisioners":

  [

    {

      "type": "shell",

      "inline": [

        "sleep 3",

        "echo \"NOZEROCONF=yes\" >> /etc/sysconfig/network",

        "adduser nutanix-admin",

        "echo 'nutanix-admin:nutanix' |chpasswd",

        "mkdir /home/nutanix-admin/.ssh",

        "chown nutanix-admin:nutanix-admin /home/nutanix-admin/.ssh",

        "chmod 700 /home/nutanix-admin/.ssh",

        "echo \"nutanix-admin ALL=(ALL) ALL\" >> /etc/sudoers"

      ]

    },

    {

      "type": "file",

      "source": "centos.json",

      "destination": "/root/centos.json",

      "source": "httpdir/centos6-ks.cfg",

      "destination": "/root/centos6-ks.cfg"

    },

Under provisioners you can see some examples of things you can do to your image. You have the ability to add users, copy in ssh keys, install and update packages and otherwise prep the image baseline exactly as you want. However, I would also caution against adding too much in the default image as individual app-specific packages can be added with your Chef, Ansible, or configuration management tool of choice after you clone your images.

Once the image build completes, you can login to your Nutanix cluster and upload your .img file with the Image Configuration service as a DISK instead of an ISO. From there, you can create a new template VM, choose to clone the disk from the Image service and power it on to verify that you have a new working template that you didn't even have to interactively install.

From there you can prep the image for mass cloning as I described here, if you didn't already perform these steps with the Packer inline provisioning, and then immediately move on to deploying and scaling out your apps.

There are not as many KVM examples in the wild as there are for AWS or vSphere, which is one of the main reasons I wrote this post, but at least there are other builds that can be adapted like the Bento boxes from Chef here: https://github.com/chef/bento

If you find this useful, I would encourage you to share out your own public packer build files if possible.

Additional links:

https://www.packer.io/docs/builders/qemu.html