vagrant with lxc and libvirt on fedora

I recently got interested in Vagrant as a means for automating setup of virtual build and test environments, especially for my Samba/CTDB development, since in particular setup of clustered Samba servers is somewhat involved, and being able to easily produce a pristine new test and development environment is highly desirable.

It took some time for me to get it right, especially because I did not choose the standard virtualbox hypervisor but wanted to stick to my environment that uses lxc for Linux and libvirt/kvm for everything else, but also to some extent because I am now using Fedora as a host and also for many Linux boxes, and I had to learn that vagrant and lxc don’t seem to play best with Fedora. Since others might profit from it as well, I thought I’d publish the results and write about them. This post is the first in a series of articles leading up an environment where vagrant up on a fedora host provisions and starts, e.g., a 3(or 4 or …)-node Samba-CTDB cluster in LXC. This first post describes the steps necessary to run vagrant with libvirt and lxc backends on Fedora 21.

Vagrant concepts

There is extensive documentation at docs.vagrantup.com, so just a few introductory words here… Vagrant is a command line program (written in ruby) that executes simple (or also more complicated) recipes to create, provision, start, and manage virtual machines or containers. The main points are disposability and reproducibility: Call vagrant up and your environment will be created out of nothing. Destroy your work environment with vagrant destroy after doing some work, and it is gone for good. Call vagrant up on the same or on a different host and there it is again in a pristine state.

Vagrantfile

The core of a vagrant environment is the Vagrantfile, which is the recipe. The Vagrantfile specifies the resulting machine by naming a virtualization provider (usually), a base image (called box in vagrant lingo) and giving instructions for further provisioning the base box. The command vagrant up executes these steps. Since the Vagrantfile is in fact a ruby program snippet, the things you can do with it are quite powerful.

Boxes

The base boxes are a concept not entirely unlike docker images. There are many pre-created boxes available on the web, that can be downloaded and stored locally, but there is also Hashicorp’s atlas, which offers a platform to publish boxes and which is directly integrated with vagrant, comparable to what the docker hub is for docker.

Providers

Vagrant was created with virtualbox as virtualization provider, but nowadays also supports docker and hyper-v out of the box. My personal work environment consists mostly of libvirt/kvm for windows and non-linux unix systems and lxc for linux, so vagrant does not immediately seem to fit. But after some research I found out that, luckily, these providers do exist externally already: vagrant-libvirt and vagrant-lxc. These and more providers like aws, vmware and native kvm can be easily installed by virtue of vagrant’s plugin system.

Plugins

Vagrant can be extended via plugins which can add providers but also other features. Two very interesting plugins that I’ve come across and would recommend to install are vagrant-cachier, which establishes a host cache for packages installed inside VMs, and vagrant-mutate, which converts boxes between different provider formats — this is especially useful for converting some of the many available virtualbox boxes to libvirt.

Provisioning

The provisioning of the VMs can be done by hand with inline or external shell scripts, but Vagrant is also integrated with puppet, ansible, chef and others for more advanced provisioning.

The vagrant command

The central command is vagrant. It has a nice cli with help for subcommands. Here is the list of subcommands that I have used most:

vagrant box      - manage boxes
vagrant plugin   - manage plugins
vagrant init     - initialize a new Vagrantfile
vagrant up       - create/provision/start the machine
vagrant ssh      - ssh into the machine as user vagrant
vagrant suspend  - suspend the machine
vagrant resume   - resume the suspended machine
vagrant halt     - stop the machine
vagrant destroy   - remove the machine completely

All data that vagrant maintains is user-specific and stored by default under ~/.vagrant.d/, e.g. plugins, required ruby gems, boxes, cache (from vagrant-cachier) and so on.

Vagrant onto Fedora

Fedora does not ship Vagrant, but there are plans to package vagrant — not sure it will make it for f22. One could install vagrant from git, but there is also an RPM on Vagrant’s download site and it installs without problems. So let’s use that for a quick start.

It is slightly irritating at first sight that the RPM does not have any dependencies though it should require ruby and some gems at the very least. But the vagrant RPM ships its own ruby and a lot of other stuff even some precompiled system libraries under /opt/vagrant/embedded, so you can start running it without even installing ruby or any gems in the system. This vagrant is configured to always use the builtin ruby and libraries, and as you will see, this does create some problems.

At this stage, you are good to go when you are using virtualbox (which I am not). For example, there is a very new fedora21-atomic image on atlas jimmidyson/fedora21-atomic. All it takes to fire up a fedora box with this base box is:

vagrant box add jimmidyson/fedora21-atomic --provider virtualbox
mkdir -p ~/vagrant/test
cd ~/vagrant/test
vagrant init jimmidyson/fedora21-atomic
vagrant up

Installing vagrant-lxc

It is actually fairly easy to install vagrant-lxc: the basic call is

vagrant plugin install vagrant-lxc

But there are some mild prerequisites, because the plugin installer wants to install some ruby-gems, in this case json. The vagrant installed from the official RPM installs the gems locally even if a matching version is present in the system, because this vagrant only ever uses the ruby stuff installed under /opt/vagrant/embedded and under ~/.vagrant.d/gems/. For vagrant-lxc, this is not a big deal, we only need to install make and gcc, because the json gem needs to compile some C file. So these are the complete steps needed to install the vagrant-lxc provider:

sudo yum install make gcc
vagrant plugin install vagrant-lxc

Afterwards, vagrant plugin list prints

$ vagrant plugin list
vagrant-lxc (1.0.1)
vagrant-share (1.1.4, system)

Now in order to make use of it, apart from having lxc installed, we need network bridges set up on the host, and we need applicable boxes. For the network, the easiest is to use libvirt network setups, since at least on Fedora, libvirt is set to default to virbr0 anyways. So my recommendation is:


sudo yum install lxc lxc-extra lxc-templates
sudo yum install libvirt-daemon-config-network

This libvirt network component can even be installed when using virtualbox, but if you are planning to use libvirt/kvm anyways, it is a perfect match to hook the lxc containers up to the same networks as the kvm machines, because they can then communicate without further ado.

The task of getting boxes is not that easy. There are really not many of them around: You can search for the lxc provider on atlas and find a dozen or so, mostly ubuntu and some debian boxes by the author of vagrant-lxc. So I needed to create some Fedora boxes myself and this was not completely trivial, in fact it was driving me almost crazy, but this is a separate story to be told. The important bit is that I succeeded and started out with Fedora 21 and 20 boxes which I published on atlas.

So to spin up a f21 lxc box, this is sufficent:

vagrant box add obnox/fedora21-64-lxc
mkdir -p ~/vagrant/lxc-test/
cd -p ~/vagrant/lxc-test/
vagrant init obnox/fedora21-64-lxc
vagrant up --provider=lxc

I find it entertaining to have sudo watch lxc-ls --fancy running in a separate terminal all the time. After bringing up the machine you can vagrant ssh into the machine and get stuff done. The working directory where your Vagrantfile is stored is bind-mounted into the container as /vagrant so you can exchange files.

Vagrant defaults to virtualbox as a provider, which is why we have to specify --provider=lxc. In order to avoid it, one can either set the environment variable VAGRANT_DEFAULT_PROVIDER to the provider of choice, or add config.vm.provider :lxc to the Vagrantfile. One can also add a block for the provider to enter provider-specific options, for instance to set the lxc container name to be used. Here is an example of a marginally useful Vagrantfile:

Vagrant.configure("2") do |config|
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
  end 

  config.vm.define "foo"
  config.vm.box = "obnox/fedora21-64-lxc"
  config.vm.provider :lxc do |lxc|
    lxc.container_name = "vagrant-test-007"
  end
  config.vm.hostname = "test-007"
  config.vm.provision :shell, inline: "echo Hello, world!"
end

Note the conditional configuration of vagrant-cachier: If installed, users of the same base box will benefit from a common yum cache on the host. This can drastically reduce machine creation times, so better make sure it is installed:

vagrant plugin install vagrant-cachier

Installing vagrant-libvirt

Now on to use vagrant with libvirt. In principle, it is as easy as calling

vagrant plugin install vagrant-libvirt

But again there are a few prerequisites and surprisingly also pitfalls. As mentioned above, the vagrant-libvirt installer wants to install some ruby gems, specifically ruby-libvirt, even when a matching version of the gem is already installed in the system. In addition to make and gcc, this plugin also needs the libvirt-devel package. Now the plugin installation failed when I tried to reproduce it on a pristine system with a strange error: The linker was complaining about certain symbols not being available:

"gcc -o conftest -I/opt/vagrant/embedded/include/ruby-2.0.0/x86_64-linux -I/opt/vagrant/embedded/include/ruby-2.0.0/ruby/backward -I/opt/vagrant/embedded/include/ruby-2.0.0 -I. -I/vagrant-substrate/staging/embedded/include -I/vagrant-substrate/staging/embedded/include -fPIC conftest.c -L. -L/opt/vagrant/embedded/lib -Wl,-R/opt/vagrant/embedded/lib -L/vagrant-substrate/staging/embedded/lib -Wl,-R/vagrant-substrate/staging/embedded/lib -lvirt '-Wl,-rpath,/../lib' -Wl,-R -Wl,/opt/vagrant/embedded/lib -L/opt/vagrant/embedded/lib -lruby -lpthread -lrt -ldl -lcrypt -lm -lc"
/lib64/libsystemd.so.0: undefined reference to `lzma_stream_decoder@XZ_5.0'
/lib64/libxml2.so.2: undefined reference to `lzma_auto_decoder@XZ_5.0'
/lib64/libxml2.so.2: undefined reference to `lzma_properties_decode@XZ_5.0'
/lib64/libsystemd.so.0: undefined reference to `lzma_end@XZ_5.0'
/lib64/libsystemd.so.0: undefined reference to `lzma_code@XZ_5.0'
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5: return 0;
6: }
/* end */

This drove me nuts for quite a while, since no matter which libraries and programs I installed or uninstalled on the host, it would still fail the same way. The explanation is that there is the system-installed lzma library that libxml2 uses and that uses symbol versioning. But the vagrant RPM ships its own copy in /opt/vagrant/embedded/lib/liblzma.so.5.0.7, so with all the linker settings to the gcc call, this supersedes the system-installed one and the symbol dependencies fail. In the end, I found the cure comparing one system that worked and another that didn’t: The gold linker can do it, while the legacy bfd linker can’t. Spooky…

So finally here is the minimal set of commands you need to install vagrant-libvirt on Fedora 21:

sudo yum install -y vagrant_1.7.1_x86_64.rpm
sudo yum install -y make gcc libvirt-devel
sudo alternatives --set ld /usr/bin/ld.gold
vagrant plugin install vagrant-libvirt

Of course, in order for this to be really useful, one needs to install libvirt properly, I do

yum install libvirt-daemon-kvm

because I want to use the kvm backend.
Afterwards, you can bring up a fedora box like this:

vagrant box add jimmidyson/fedora21-atomic --provider libvirt
mkdir -p ~/vagrant/test
cd ~/vagrant/test
vagrant init jimmidyson/fedora21-atomic --provider libvirt
vagrant up

As already mentioned it is a good idea to install the vagrant-cachier and vagrant-mutate plugins:

vagrant plugin install vagrant-cachier
sudo yum install -y qemu-img
vagrant plugin install vagrant-mutate

With the mutate plugin you can convert some of the many virtualbox boxes to libvirt.

For the fun of it, here is the vagrantfile, I used to develop and verify the minimal installation procedure inside vagrant-lxc… ;-)

VAGRANTFILE_API_VERSION = 2 

INSTALL_VAGRANT = <<SCRIPT
set -e
yum install -y /vagrant/vagrant_1.7.1_x86_64.rpm
yum install -y make gcc
sudo -u vagrant vagrant plugin install vagrant-lxc
yum install -y libvirt-devel
alternatives --set ld /usr/bin/ld.gold
sudo -u vagrant vagrant plugin install vagrant-libvirt
sudo -u vagrant vagrant plugin install vagrant-cachier
yum install -y qemu-img
sudo -u vagrant vagrant plugin install vagrant-mutate
sudo -u vagrant vagrant plugin list
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
  end 

  config.vm.define "vagrant-test" do |machine|
    machine.vm.box      = "obnox/fedora21-64-lxc"
    machine.vm.hostname = "vagrant-test"
    machine.vm.provider :lxc do |lxc|
      lxc.container_name = "vagrant-test"
    end
    machine.vm.provision :shell, inline: INSTALL_VAGRANT
  end
end

Summary

So now we are in the position to work with vagrant-lxc and vagrant-libvirt on fedora, and we also have fedora lxc boxes available. I am really intrigued by the ease of creating virtual machines and containers. From here on, the general and provider-specific guides on the net apply.

Here is a mock-up transcript of the steps taken to set up the environment:

# use current download link from https://www.vagrantup.com/downloads.html
wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.2_x86_64.rpm
sudo yum install vagrant_1.7.2_x86_64.rpm
sudo yum install make gcc libvirt-devel qemu-img
sudo yum install lxc lxc-extra lxc-template
sudo yum install libvirt-daemon-kvm

vagrant plugin install vagrant-lxc
vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-mutate

vagrant box add obnox/fedora21-64-lxc
vagrant box add obnox/fedora20-64-lxc
vagrant box add jimmidyson/fedora21-atomic --provider libvirt
vagrant box add purpleidea/centos-7.0
vagrant box add ubuntu/trusty64
vagrant box repackage ubuntu/trusty64  virtualbox 14.04
mv package.box trusty64.box
vagrant mutate trusty64 libvirt
...

What next?

I will post follow-up articles about the problems creating Fedora-lxc-boxes and about the general Vagrantfile mechanism for bringing up complete samba-ctdb clusters. I might also look into using vagrant from sources, potentially looking into fedora packaging, in order to circumvent the discomfort of running a bundled version of ruby, the universe, and everything… ;-)

8 comments on “vagrant with lxc and libvirt on fedora

  1. Ernad Husremovic on said:

    Great post.
    I await for next posts on this topic eagerly.

  2. Jason Brooks on said:

    Great post! Keep an eye out for: https://github.com/pradels/vagrant-libvirt/issues/290#issuecomment-68797403

  3. Josef Stribny on said:

    Please note I created coprs for vagrant with libvirt provider:

    f20: https://copr.fedoraproject.org/coprs/jstribny/vagrant-f20/
    f21: https://copr.fedoraproject.org/coprs/jstribny/vagrant-f21/
    f22: https://copr.fedoraproject.org/coprs/jstribny/vagrant-f22/
    vagrant1 scl for RHEL: https://copr.fedoraproject.org/coprs/jstribny/vagrant1

    To install it on Fedora it’s enough to add the Copr repository and run ‘yum install vagrant-libvirt’. Also they all use the system RPMs and do not bundle anything.

    • obnox on said:

      Oh, that is looks really cool. I have to test it!
      I have to admit that I am rather new to fedora and wasn’t aware of these repositories… :-)
      Is that basically the package from the fedora packaging review BZ
      https://bugzilla.redhat.com/show_bug.cgi?id=1020456 ?
      We should get it into f22…

    • Casey Link on said:

      For me these packages installed fine, but vagrant didn’t recognize that the lxc and libvirt plugins were installed.

      Following obnox’s instructions here worked however.

  4. Casey Link on said:

    Also for those who want to run vagrant on a user without general sudo permission, or don’t want to be prompted to type their sudo password, here’s the sudoers perms you need: https://gist.github.com/Ramblurr/3fbc11f797e61f0b1a03

  5. Erik Van Kelst on said:

    Amazing man, thanks a lot!
    I’ve struggled to accomplish the exact same thing on vanilla ubuntu-14.04, ubuntu-15.04, fedora-20, fedoea-21 boxes without installing the whole ruby dependencies..
    You just saved my day!

    Erik