Running puppet-keystone spec tests in docker container

I am working on adding features to puppet-keystone such as support for configuring Keystone to use the LDAP identity backend, read-only as well as read-write, and to have multiple LDAP identity backends.  I wanted to easily run tests on different platforms using docker.

I started with a fedora 20 docker image, and installed the ruby and ruby bundle packages.  I called this image "puppet-keystone".  In order to share my development source code working directory, I mount my home directory inside the container as "/share".  In order for the container to have access, I need to create my user and group inside the container:

# groupadd -g 1000 rmeggins
# useradd -g rmeggins -u 1000 rmeggins

I ran docker like this:

$ sudo docker run -v /home/rmeggins:/share -i -t -u 1000 puppet-keystone

However, I could not see /share - permission denied.  Not sure why, but I figured that I had to run the container with --privileged like this:

$ sudo docker run --privileged -v /home/rmeggins:/share -i -t -u 1000 puppet-keystone

This allowed me full access to the /share volume.

Then I ran the test setup:

$ cd /share/puppet-keystone
$ GEM_HOME=vendor bundle install
$ GEM_HOME=vendor bundle exec rake spec SPEC_OPTS="--format d"

This caused errors reading from manifests/wsgi/apache.pp:

  1) keystone::wsgi::apache on RedHat platforms configures apache serving keysto
ne with mod_wsgi
     Failure/Error: it { should contain_service('httpd').with_name(platform_para
meters[:httpd_service_name]) }
       invalid byte sequence in US-ASCII at /share/puppet-keystone/spec/fixtures
/modules/keystone/manifests/wsgi/apache.pp:1 on node
     Shared Example Group: "apache serving keystone with mod_wsgi" called from .

Sure enough, file reports a different type for that file:

$ find manifests -name "*.pp" | xargs file
manifests/init.pp:                      ASCII text
manifests/endpoint.pp:                  C++ source, ASCII text
manifests/python.pp:                    C++ source, ASCII text
manifests/service.pp:                   C++ source, ASCII text
manifests/cron/token_flush.pp:          C++ source, ASCII text
manifests/wsgi/apache.pp:               C++ source, UTF-8 Unicode text

For some reason, inside the container, it can't read this file, but I don't have this problem from the host.  I was able to work around the problem in the container by running the test like this:

$ LC_CTYPE=C.UTF-8 LANG=C.UTF-8 GEM_HOME=vendor bundle exec rake spec SPEC_OPTS="--format d"

Now I am able to run the tests successfully with no errors

How to disable host firewall for guest virtual machines

I'm using Fedora 20. I was having quite a bit of trouble mounting NFS host export directories in my virtual machine guests. It seemed to be firewall related. If I shut off the firewall completely
  # systemctl stop firewalld.service

then NFS would work, but the VM would lose the ability to do networking with the public network e.g. yum updates would fail. I could not seem to get the magic combination of firewall ports/services to allow. Even after allowing NFS, RPC bind, and port 20048 tcp and udp, NFS mounts would still fail.

I finally figured out that I needed to simply disable the firewall on the virtual interface.

Step 1: Find out the name of the virtual bridge device:
  # ip addr
  4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:54:00:27:a8:fb brd ff:ff:ff:ff:ff:ff
    inet brd scope global virbr0
       valid_lft forever preferred_lft forever

The name is virbr0

Step 2: Tell firewalld to trust any connection using this device:
  # firewall-cmd --zone=trusted --add-interface=virbr0
  # firewall-cmd --permanent --zone=trusted --add-interface=virbr0

This tells firewalld to allow any/all traffic using the virtual bridge. Using --permanent tells firewalld to make this configuration persistent.

Using Windows users in OpenStack via FreeIPA, LDAP, and Cross Domain Trust

In part 1, the focus is on the automated set up of FreeIPA, Active Directory, and Red Hat OpenStack 4, in Fedora 20, Windows Server 2008, and RHEL 6.5 respectively. The demo uses FreeIPA running on Fedora 20, but Red Hat Enterprise Linux version 7 (RHEL 7) with the built-in Identity Management (IdM) could be used just as well.

This part will show how to put them all together to use a Windows user in OpenStack via FreeIPA LDAP and cross domain trust. Here is the demo video:

What follows below will explain how the Windows users are made available in OpenStack.

Here is the initial OpenStack user list, with the service accounts.

OpenStack Keystone has been set up to use the LDAP Identity backend, using FreeIPA as the LDAP server. These service accounts have entries in FreeIPA, which are shown in the FreeIPA user list.

This FreeIPA has been set up with cross domain trust with the Windows domain ad1dom.test.

FreeIPA supports a "compatibility" subtree. This provides a sort of "virtual view" of data in the server in another suffix, and also, with trusts, data that exists outside of the server. For trust accounts, the trust code makes these entries look like posixAccount entries, and adds uidNumber, gidNumber, etc. values to make them look like a real POSIX user accounts. This is the cn=compat subtree. User entries are in the cn=users subtree. In the demo, the full suffix is cn=users,cn=compat,dc=ipa1dom,dc=test. This shows the output of the ldapsearch command:

ldapsearch -LLL -b cn=users,cn=compat,dc=ipa1dom,dc=test 'objectclass=posixAccount' dn

To show a Windows user account being used in OpenStack, the demo creates a Windows user account.

In order to use this account with OpenStack, it has to first be "pulled" into FreeIPA. Trusts are enabled in FreeIPA with the ipa-adtrust-install command. The option --enable-compat allows user accounts from trusted domains to show up as "virtual" entries in the cn=compat tree. When the trust is set up with the remote domain using ipa trust-add $otherdomain, this allows entries in that domain to be "viewed" in the cn=compat tree. FreeIPA will not automatically "pull" the entries from the trusted domain into the LDAP compat tree. In order for a trusted account to have a virtual entry in the compat tree, the entry must be requested explicitly by user ID. With trusts, this means to do an LDAP search with a search filter like (uid=$remoteuserid@$remotedomain). In the demo, this is (uid=openstack@ad1dom.test). When this user ID is searched for, FreeIPA will create a virtual entry in the cn=compat tree with the POSIX attributes.

In the demo, Keystone is configured to use uid for the user ID attribute. Many of the user specific Keystone commands will do an LDAP search like the above. The demo shows the use of the keystone user-role-add command, which will do the above LDAP search, and also enable the user account for Keystone with a tenant and a role. The command keystone user-get will also do the same LDAP search. In the demo, the keystone user-role-add command is used because it will both pull the AD account into the FreeIPA cn=compat tree and make it available via LDAP, and also enable the user for use in Keystone and OpenStack.

The Windows user can be used to login to the OpenStack dashboard,

and used to start a virtual machine instance.

The demo then shows that the Windows user can be used with the Nova command line client.

If we do the ldapsearch -LLL -b cn=users,cn=compat,dc=ipa1dom,dc=test 'objectclass=posixAccount' dn again in FreeIPA, we can see that the openstack@ad1dom.test user now has an entry in the compat tree.


The virtual entries in the cn=compat tree in LDAP are not persistent. If the FreeIPA LDAP server is restarted, the user entries from the trusts will disappear, and must be searched for explicitly again in order to make them available via LDAP. From an OpenStack perspective, this will cause the trusted users to disappear from user lists e.g. they will not be in the list of users in the dashboard, nor in the output of the keystone user-list command. Using OpenStack, the easiest way to make them available in LDAP again is to use the keystone user-get command - for example

keystone user-get openstack@ad1dom.test

The users from the trusted domain are stored in sssd. If the connection with Windows is lost (network, machine is down), the users will still be available from sssd.

Integration of FreeIPA, Active Directory, and OpenStack Horizon - part 1


One of the newer features of FreeIPA is the ability to set up a cross-realm trust with Windows Active Directory (AD). This allows Windows users to authenticate directly to Linux machines. This is the preferred long-term solution, to replace the current Windows Synchronization/Password Synchronization solution. Horizon is an OpenStack project that provides a web based GUI "dashboard" for other OpenStack components. I will outline the steps necessary to create Windows users and authenticate to Horizon with those user credentials, via FreeIPA.


Most of the below steps can be automated. I have started a github project called ipa-ad-trust-demo which contains the scripts and config files used. I have tested the script on Fedora 20 using libvirt/kvm/qemu for virtualization. The script depends on two other of my github projects:

  • auto-win-vm-ad - This handles the creation of the Windows VM, the unattended set up of an Active Directory Domain, unattended set up of MS Certificate Services, and DNS configuration. The entry point is the shell script

  • - This is just a wrapper around virt-install, kickstart, and cloud-init that simplifies virtual machine creation using shell script based config files.

The demo is run using the script like this:
$ cd /path/to/ipa-ad-trust-demo
$ PATH=$PATH:/path/to/scripts:/path/to/auto-win-vm-ad:. ./ global.conf ad1.conf ipa1.conf

At the end of the run, you will have a Windows VM running AD, and a Fedora 20 VM running FreeIPA. Windows users will be able to authenticate to FreeIPA. The script demonstrates an AD user doing a kinit against FreeIPA, and also demonstrates using those credentials to do ldapsearch -Y GSSAPI to search against AD.

Automation Pre-requisites

You will need the following packages:
# yum -y install unar qemu-img qemu-kvm libvirt virt-manager libguestfs-tools

You will need to manually download the Fedora 20 cloud image from and put it in your libvirt images directory (/var/lib/libvirt/images):
# cd /var/lib/libvirt/images
# wget

Then edit ipa1.conf to use this as the VM_DISKFILE_BACKING. However, if you plan to do many iterations of testing/vm creation/vm destruction, I strongly encourage you to first create a VM using the standard image, then update it, then install all of the packages required for FreeIPA and trust (and optionally the GNOME Desktop), and use this pre-prepared image as your VM_DISKFILE_BACKING image. See the Fedora 20 and FreeIPA section below for more information. The difference is stark - using the standard image, it takes about 100 minutes - using a pre-prepared image, it takes about 8 minutes.

Both and should be in the PATH. Using "." in PATH allows the config files to be sourced as
. $file

where $file is something like global.conf rather than ./global.conf or /path/to/global.conf.

The script should be run as a regular user, not root, and the system should be set up to allow this user to use sudo with no password access e.g. in /etc/sudoers:

You should set SUDOCMD=sudo in global.conf to use sudo. The use of SUDOCMD instead of hard-coding "sudo" allows the use of sudo flags or even other commands.

The settings in global.conf apply to all VMs. This where to set things like the virtual network name, IP address base/range, VM disk image directory, and SUDOCMD.

Automation Notes

The script does all of the setup work - ipa setup, trust setup with AD, testing trust using kinit and ldapsearch -Y GSSAPI, etc. This is launched as a cloud-init runcmd. By default, ipa-server-install and other ipa command line tools do not work from a cloud-init runcmd due to SELinux. This is mostly due to the fact that the runcmd runs with an unexpected context cloud_init_t. After the first run which failed, I used audit2allow to create the cloudinit.pp module, which is now loaded before doing ipa-server-install.

ipa-adtrust-install will restart 389, which causes named to lose the connection, and by default it will not attempt to reconnect for 60 seconds. The script has a 60 second sleep here which should be sufficient time.

Windows and Active Directory

For manual setup/install, you must install Windows and set up Active Directory first, before installing and setting up FreeIPA. Otherwise, you will have to coordinate so that ipa-adtrust-install and ipa trust-add are not run until after AD is available.

Microsoft provides virtual disk images of Windows Server 2008. NOTE: These images are to be used for evaluation and demo purposes only. DO NOT use them for any other purpose, especially in production. The EULA for the evaulation images is quite specific about this. The FreeIPA website documents the procedure for downloading the images, converting to qemu/kvm/RHEV format (qcow2), creating a VM, and setting up an Active Directory domain, using the GUI.

Finally, follow the Windows steps listed at

Windows Notes

For all of the gory details about how works, see The script also works with ISOs, but the script expects to use the evaluation disk images.

I was not able to use the windows image as VM_DISKFILE_BACKING in conjunction with VM_WAIT_FILE. It seems that writing to the disk image file somehow corrupts the Windows registry, in the SOFTWARE hive, that causes virt-ls and virt-cat to fail. The way that the script detects that the Windows VM setup is complete is by writing a file to C:\ (VM_WAIT_FILE), and using virt-cat to test for existence of the file. The script just makes a copy of the original Windows disk image for the new VM. This is slower but more robust.

Fedora 20 and FreeIPA

I use the Fedora 20 cloud image provided by Fedora. This is cloud-init enabled which makes it easy to do automated install and setup. The demo uses FreeIPA running on Fedora 20, but Red Hat Enterprise Linux version 7 (RHEL 7) with the built-in Identity Management (IdM) could be used just as well. Note that the image by now is quite old (in Fedora standards), so if you plan to do a lot of iterative testing (e.g. create vm/test/destroy vm/repeat) with this image, you should plan to first install a "base" VM, then do a yum update, plus install all of the FreeIPA packages necessary:
# yum -y update
# yum -y install freeipa-server freeipa-server-trust-ad \
  bind-dyndb-ldap bind bind-utils samba4-winbind-clients \
  samba4-winbind samba4-client openldap-clients rng-tools

In order to get a graphical desktop (e.g. if you want to run Horizon in a VM), you will also need to:
# yum -y groupinstall "GNOME Desktop"

Then follow the instructions about how to enable graphical login.

FreeIPA set up requires a lot of entropy from /dev/random. It is possible for ipa-server-install to hang during Kerberos server setup. To avoid this issue, create the VM with a virtual RNG e.g.
$ sudo virt-install --rng /dev/random --name ipa ...

This should create the /dev/hwrng device in the VM. Then, in the VM, install the rng-tools package, and make sure to run rngd -r /dev/hwrng before running ipa-server-install. This should supply plenty of entropy to /dev/random in the VM from the host, and avoid the hang, and make setup run much faster. The script takes care of this.

Announcing 389 Directory Server 1.2.10 Release Candidate 1 - Testing

1.2.10 Release Candidate 1 was released to the Testing repos in Fedora and EPEL. RC1 has no new features since alpha 8, but many, many bug fixes.

We've also stopped using Red Hat Bugzilla for our 389 issue tracking, and are now using Trac. You'll need a Fedora account in order to file new tickets.

Release Notes -