During the day I’m a manager of one of the greatest security teams on the planet (in my biased estimation), but at night (and random times throughout the day), I’m a sysadmin tinkerer. There’s just something about goofing off with operating systems that appeals to me; this is likely what caused me to devote five years of my life to working on Annvix back in the day.
I’ve been running a local IPA install for quite a few years, but because you really need IPA to run on its own dedicated system (and given I have enough machines running in this house) I’ve been using KVM to handle the virtualization of an IPA server. IPA is really cool and allowed me to discard my homegrown LDAP+Kerberos setup for something with some enterprise gusto to manage authentication, identification, and authorization policies for my home network (insert obligatory comment about overkill here). I started using IPA on CentOS 6 and a year ago moved both guest and host to CentOS 7 which is working pretty good other than, for some odd reason, python is randomly segfaulting at times. I don’t know what the cause is, I’ve filed abrt reports, and it’s a concern when the language that your system updater is based on decides to start crashing before installing updates or, even worse, during the installation of updates and making a mess of things (this does not make RPM happy!). The oddest thing is that none of the other CentOS 7 systems (on bare metal) exhibit this behaviour.
The other problem is if my IPA server decides to tip over, I don’t have a failover setup (again this being at home). So while I was thinking about the best way to stand up another IPA server as a replicating slave to then promote it to the master and migrate away from whatever is causing all these nasty segaults, I was reminded by FreeNAS that an update was available for it, and I started thinking about jails and whether they would run Linux. After starting down the rabbit trail, I found out about iohyve which is a FreeBSD (what FreeNAS is based on) bhyve manager. Byhve is a hypervisor that runs on FreeBSD, basically like KVM (which I’ve been using) or OpenVZ (which I’ve used with VPS hosting), or Xen.
So bhyve is for FreeBSD what KVM is for Linux. And this is where the sysadmin/tinkerer/geek in me thinks “cool” and away disappears a weekend.
For the purposes of this post/tutorial, you need to be running FreeNAS 9.10 (you can probably do this easily enough with FreeBSD, but I’ve not tried). There is also documentation on Using iohyve.
From your FreeNAS system you need to know your ethernet interface name (in the web UI go to Network -> Network Interfaces, in my case em0) and the storage pool name (Storage -> Volumes, in my case the pool is named storage). The actual setup of iohyve needs to be done as root over SSH, so you’ll need that running as well.
As root, we need to create the environment iohyve requires. I used the following commands to create the pool for its use:
# iohyve setup pool=storage kmod=1 net=em0
Setting up iohyve pool...
On FreeNAS installation.
Checking for symbolic link to /iohyve from /mnt/iohyve...
Symbolic link to /iohyve from /mnt/iohyve successfully created.
Loading kernel modules...
bridge0 is already enabled on this machine...
Setting up correct sysctl value...
net.link.tap.up_on_open: 0 -> 1
This tells iohyve to install the required ZFS datasets and kernel modules. We use kmod=1 to tell iohyve to load the required kernel module, pool=storage tells it which pool to use for files (in this case, storage) and net=em0 sets up the network bridge to this interface (iohyve can only be bound to a single interface). You can use multiple pools for iohyve, however I only have one pool on the system.
Next, you need to create a few tunables in FreeNAS. Heading back to the web UI, go to System -> Tunables and create the following two tunables:
- variable: iohyve_enable, values: YES, type: rc_conf
- variable: iohve_flags, values: kmod=1 net=em0, type: rc_conf
The iohyve_enable variable tells FreeNAS to load iohyve support at boot, and the iohyve_flags are the same kmod and net options we used when setting up iohyve initially.
The next step is to download an ISO image for iohyve to use for installing a virtual machine. In my case, I want to run CentOS 7. There are plenty of mirrors to choose from for the minimal ISO which is probably what you want since you can install any specific software required using yum after it’s installed.
# iohyve fetch http://centos.mirror.iweb.ca/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso
Fetching http://centos.mirror.iweb.ca/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso...
/iohyve/ISO/CentOS-7-x86_64-Minimal-1611.iso/C100% of 680 MB 9 MBps 01m08s
You need to use iohyve to fetch the ISO image (somewhat annoyingly) so you can’t just copy an existing ISO over (although I believe you can do a fetch over NFS or provide it locally via HTTP from another system on your network).
Once you have the ISO downloaded, you can configure a new virtual machine. Check to make sure you have the ISO available:
# iohyve isolist
Listing ISO's...
CentOS-7-x86_64-Minimal-1611.iso
Then, to create the machine with 20GB of space (the same size as my existing KVM machine for IPA, which is more than enough):
# iohyve create ipa-slave 20G
Creating ipa-slave...
# iohyve list
Guest VMM? Running rcboot? Description
ipa-slave NO NO NO Sun Mar 12 16:39:10 MDT 2017
Now you can configure the specifics of the machine:
# iohyve set ipa-slave ram=1G cpu=1 os=custom loader=grub-bhyve
Setting ipa-slave ram=1G...
Setting ipa-slave cpu=1...
Setting ipa-slave os=cuscom...
Setting ipa-slave loader=grub-bhyve...
This sets my virtual machine to have 1GB of RAM, use one virtual CPU, use the “custom” operating system type (we need this later, even though we will be using CentOS 7), and uses the grub-bhyve loader which is required by Linux guests. The iohyve wiki has more details on operating system types and which values to use depending on which Linux operating system you intend to install.
When using a CentOS 7 guest, iohyve currently cannot boot from an XFS partition (which is the default), and due to the limitations of the commandline installer, we can’t tell Anaconda to use something other than XFS. Another thing I found, with some trial and error, is you want to use traditional partitions and not the LVM-based partition scheme (so plan out your filesystem in advance to ensure you have enough size!). This is the main reason for using the “custom” operating system type. We’ll fix that later.
To work around this, we’ll use a simple kickstart file to get us to a minimal working system from which we can install the rest of what we want.
In order to make grub boot and use the kickstart file, you need to edit /iohyve/ipa-slave/grub.cfg so it looks like:
linux (cd0)/isolinux/vmlinuz inst.ks=http://somewhere.internal/ks.cfg
initrd (cd0)/isolinux/initrd.img
boot
and the ks.cfg file would look something like (see the documentation for more info):
#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text install
text
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8
# Network information
network --bootproto=static --device=eth0 --gateway=192.168.1.1 --ip=192.168.1.16 --nameserver=192.168.1.1 --netmask=255.255.255.0 --noipv6 --activate
network --hostname=ipa-slave.mydomain.com
# Root password
rootpw --iscrypted $6$/GdEAa2DwhlmU.Vr$R/L.fEc6QwtFiTMLd04HR1SuS7NrsdA.NuQyQ17RbBk8p37oGD/hVvRIOw0v5x6pSC6uU4NigueNmEXvQ8pzo0
# System services
services --enabled="chronyd"
# System timezone
timezone America/Edmonton --isUtc --ntpservers=ntp.mydomain.com
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --drives=sda --all
# Disk partitioning information
autopart --type=plain --fstype="ext4"
%packages
chrony
%end
%addon com_redhat_kdump --disable --reserve-mb='auto'
%end
Before starting the installation, make sure you can retrieve that file with something like curl. Next, start the installation:
# iohyve isolist
Listing ISO's...
CentOS-7-x86_64-Minimal-1611.iso
# iohyve install ipa-slave CentOS-7-x86_64-Minimal-1611.iso
Installing ipa-slave...
GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ipa-slave in second terminal to complete GRUB process...
From another terminal, ssh into your FreeNAS server again in order to connect to the serial console by using:
# iohyve console ipa-slave
Starting console on ipa-slave...
~~. to escape console [uses cu(1) for console]
Connected
...
Starting installer, one moment...
anaconda 21.48.22.93-1 for CentOS Linux 7 started.
* installation log files are stored in /tmp during the installation
* shell is available on TTY2
* when reporting a bug add logs from /tmp as separate text/plain attachments
14:21:39 Not asking for VNC because of an automated install
14:21:39 Not asking for VNC because text mode was explicitly asked for in kickstart
Starting automated install..
...
Now sit back while it automatically installs. This will be quite the minimal install, however it will get you up and running with an ext4-based system that iohyve can boot up, and from there you can install individual packages or package groups.
When the install is complete, you will see something like this on the console:
[ OK ] Started Restore /run/initramfs.
[ OK ] Reached target Shutdown.
dracut Warning: Killing all remaining processes
Rebooting.
[ 819.859124] Restarting system.
However, it’s not actually restarting the system. Switch back to the other console and you will see:
Unhandled ps2 keyboard command 0xf6
# iohyve list
Guest VMM? Running rcboot? Description
ipa-slave YES NO NO Sun Mar 12 16:39:10 MDT 2017
As you can see from the list command, the virtual machine is not running even though it said it was restarting. In order to start the machine, you must use:
# iohyve start ipa-slave
Starting ipa-slave... (Takes 15 seconds for FreeBSD guests)
[root@heimdall] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ipa-slave in second terminal to complete GRUB process...
Now switch back to the original console (that you’ve not disconnected) and you will see the system booting. I had some difficulty with dracut-initqueue warning about incessant timeouts, so it took some time for the system to boot and even then I ended up in a rescue shell.
As annoying as this is, it’s not too terribly difficult to solve although I hate how hackish it needs to be. The root of the problem seems to be that dracut wants to connect to this kickstart file before the network is up. So we need to do some chroot shenanigans and fix the initramfs:
# mkdir /mnt
# mount /dev/sda3 /mnt
# mount /dev/sda1 /mnt/boot
# chroot /mnt
# cd /boot
# cp initramfs-3.10.0-514.el7.x86_64.img initramfs-3.10.0-514.el7.x86_64.img.bak
# dracut -f /boot/initramfs-3.10.0-514.el7.x86_64.img 3.10.0-514.el7.x86_64
# ls -al initramfs*
-rw------- 1 root root 45180381 Mar 14 08:24 initramfs-0-rescue-715baff9360a47a89af2ddbc55b9f0cf.img
-rw------- 1 root root 44766225 Mar 14 09:26 initramfs-3.10.0-514.el7.x86_64.img
-rw------- 1 root root 17437682 Mar 14 09:25 initramfs-3.10.0-514.el7.x86_64.img.bak
Judging by the difference in size between the rescue image, the newly created image, and the previous one that was copied, this is a pretty strong indication that something is missing.
Also, there are a few more steps to do before we can get CentOS 7 booted properly. The first is to edit /iohyve/ipa-slave/grub.conf and remove the kickstart reference. I’m not 100% sure this is required after we perform the next action, but after a lot of beard-tugging (and to spare you the same) I’m suggesting you remove it. Also, you need to set the operating system type back to CentOS 7:
# iohyve set ipa-slave os=centos7
Setting ipa-slave os=centos7...
If you’ve stopped the virtual machine, great, if it’s still waiting for network timeouts, you can forcibly stop it with iohyve destroy ipa-slave and then we can use iohyve start ipa-slave to start it back up again (this from the session not attached to the console, of course).
Now on the console, you should be able to watch the virtual machine boot and arrive at a login prompt.
Once you login, you probably want to install a few other things given we opted for the minimal install:
# yum update -y
# yum install net-tools vim-enhanced zsh ipa-server
# systemctl status sshd
The last is to make sure that sshd is running so you can ssh in and carry on (at least for me, an 80x25 console is pretty darn tiny). I also prefer the enhanced vim, and having tools like ipaddr and ifconfig are just plain old handy, and of course the whole point of this exercise was to set this up as an IPA server.
Finally, once you verify you can ssh into the server, disconnect from the console by typing the tilde and CTRL-D (so ~ + CTRL-D).
To finish up, let’s give the virtual machine a decent description and tell it to start at boot:
# iohyve set ipa-slave description="IPA Slave server"
Setting ipa-slave description=IPA Slave server...
# iohyve set ipa-slave boot=1
Setting ipa-slave boot=1...
# iohyve list
Guest VMM? Running rcboot? Description
ipa-slave YES YES YES IPA Slave server
At this point, you can make a backup or snapshot of the virtual machine (which is one of the first things I wanted to do after all of the effort of figuring out the above!):
# iohyve snap ipa-slave@base-install-20170314
Taking snapshot ipa-slave@base-install-20170314
# iohyve snaplist
ipa-slave@base-install-20170314
Currently there doesn’t seem to be a way to remove snapshots.
I haven’t played with it enough to know whether or not the performance is better than KVM on Linux, but I enjoyed fiddling with this to get it figured out and working. Hopefully this is helpful to others; there have been quite a few references to the desire to run CentOS 7 on FreeNAS using iohyve, but even the upstream site indicates it is currently not possible (although according to this comment, it’s on the roadmap). With a bit of fiddling, it is possible.
The next step now is to figure out how to get an IPA replication slave setup because the documentation is not intuitive at all and setting up a replicated slave is about the only way to migrate IPA from one machine to another. Wish me luck, and I hope this is helpful to people interested in running CentOS 7 with iohyve.