New project: Nunc Stans

The poll() based event framework for 389 is being revamped to use a new framework based on the project Nunc Stans.  Nunc Stans is a thread pool based event framework wrapper.  That is, it provides a layer on top of event frameworks like libevent, tevent, libev, etc., or directly on top of epoll() (if we didn't want to use an event framework).  Nunc Stans has some features that are interesting to 389:

  • It provides thread safe access to the event framework, allowing the use of non-thread safe event frameworks.

  • It provides timed I/O events to facilitate idle timeouts.

  • It provides the ability to run event callbacks in a thread pool or in the main event loop thread.

  • It uses lock free data structures to reduce thread contention.

The project is hosted at
Information about how 389 plans to use nunc-stans is here

Using Keystone with HTTPD vs. the openstack-keystone service

By default, the Red Hat/Fedora/RDO openstack-keystone package provides a systemd service file openstack-keystone.service for managing the eventlet-based Keystone daemons. If you configure Keystone to use HTTPD via Apache modwsgi, the openstack-keystone.service no longer works, and you get strange errors if you try to use it:
 # systemctl status openstack-keystone
 Failed: error ....
 # systemctl start openstack-keystone
 Failed: port is in use: 35357
 Failed: port is in use: 5000
 ... other errors ...

See for more details.

The service doesn't know that Keystone is no longer a standalone service, but a webapp controlled by httpd. You can use this "trick" to "alias" openstack-keystone to httpd:
 # ln -s /usr/lib/systemd/system/httpd.service /etc/systemd/system/openstack-keystone.service
 # systemctl daemon-reload

This will override the /usr/lib/systemd/system/openstack-keystone.service provided with the openstack-keystone package. Now, when you execute a command related to openstack-keystone, it will instead be redirected to httpd, and you will see the status of httpd instead.

Aligning puppet class parameters with Emacs

Puppet is a bit of a stickler about making sure class parameter => value statements line up correctly like this:
  my_resource { 'resource_name':
   short_name            => 'value',
   a_very_very_long_name => 'value',
   another_name          => 'value,

It can become very tiresome when adding a new parameter inside a long list of class parameters to have to go back and line up all of those =>. Fortunately, I use Emacs to edit Puppet code, so I can use the align-regexp function:

  • Highlight the region you want to align
  • Type M-x align-regexp
  • Type =>

et voila! Your class parameter assignments are nicely aligned. Thanks Emacs!

Monitoring epoll events

When doing analysis of daemon processes that use epoll, you may want to take a peek inside the epoll structure. This is available via the /proc/$PID/fdinfo.

  1. Find the process id (PID) of the daemon process e.g. # pidof procname

  2. Find the epoll file descriptor (fd) e.g. # epollfd=`ls -al /proc/$PID/fd | awk '/eventpoll/ {print $9}'`

  3. Dump the information about the epoll fd e.g. # cat /proc/$PID/fdinfo/$epollfd

The information looks like this:
pos:    0
flags:  02000002
tfd:        6 events:       19 data:                6
tfd:       20 events:       19 data:               14
tfd:       21 events:       19 data:               15
tfd:     2446 events:       19 data:              98e
tfd:     1991 events:       19 data:              7c7
tfd:     1065 events:       19 data:              429
tfd:     2444 events:       19 data:              98c

Each tfd is the decimal number of the file descriptor being polled by epoll. The events field is the hex value of the flags specified in the field when calling epoll_ctl. 0x19 is EPOLLIN|EPOLLERR|EPOLLHUP. The data field is the hex value of the tfd decimal value.

If you suspect that your application is leaking file descriptors, and you believe the problem is that the application is adding fds to epoll but never deleting them, you can periodically poll the epoll fdinfo, to look for descriptors that are persistent. You should first figure out which file descriptors your application uses as persistent descriptors e.g. the listener sockets or the eventfd notifier.

Using inotifywait to compress 389 log files

inotifywait is used to efficiently poll for changes to a file or directory. On Red Hat/Fedora family operating systems, this is provided by the inotify-tools package - yum install inotify-tools. inotifywait can be used to watch for rotated access log files to compress. For example, if your access logs grow large due to a lot of traffic, you may want to compress the files to save disk space. In my testing, I have found that compressing an access log file using bzip2 reduces the size to about 3% of the original size. When the directory server rotates a log file, it issues a rename(), then recreates the access log file "access" again. You can use a shell script like this to monitor and compress rotated access log files:
# monitor and compress access logs
mon_and_comp_logs() {
    # the server does rename access -> access.20..... when rotating, then
    # creates a new file named "access"
    inotifywait -e move /var/log/dirsrv/slapd-INST --exclude '.bz2$' -m | while read dir act name ; do
        case "$file" in
            echo compressing $file
            bzip2 $file &

Then run this in the background:
# mon_and_comp_logs > mon_and_comp_logs.log 2>&1 &

How to set AD password from command line


  • ldapsearch, ldapmodify - openldap-clients package
  • iconv - glibc-common package
  • base64 - coreutils package


Let's say you want to use ldapmodify to modify the password of a user in AD. You'll first need the full DN of the user:
# ldapsearch -xLLL -H ldap://ad.example.test \
 -D "cn=administrator,cn=users,dc=example,dc=test" -W \
 -b "cn=users,dc=example,dc=test" "samaccountname=username" dn

Next, encode the password. Note that the literal double quotes are required. Also, don't forget to use echo -n or your password will end with a newline:
# b64pwd=`echo -n \""thepassword"\"|iconv --to utf-16le|base64`

Next, set the password. You will have to use either real TLS with a real CA cert from AD:
# LDAPTLS_CACERT=/path/to/ad-ca.pem ldapmodify -xLLL -ZZ -H ldap://ad.example.test \
 -D "cn=administrator,cn=users,dc=example,dc=test" -W <<EOF
dn: full dn of user from above
changetype: modify
replace: unicodePwd

OR - for quick&dirty testing:
# LDAPTLS_REQCERT=never ldapmodify -xLLL -ZZ -H ldap://ad.example.test \
 -D "cn=administrator,cn=users,dc=example,dc=test" -W <<EOF
dn: full dn of user from above
changetype: modify
replace: unicodePwd

Note that you use unicodePwd:: with two colons because the value is base-64 encoded

How to configure OpenStack Keystone with users in non-default domain

OpenStack Keystone Juno version supports multiple domains. One of the common use cases is the ability to use the identity store already existing in the enterprise (e.g. an LDAP server), so that existing users can be assigned Keystone roles to do things like login to the dashboard/Horizon, launch machines, etc. In addition, the ability for Keystone to store the service user accounts (each OpenStack service has an associated "user" account) in its default sql database so as not to "pollute" the identity store in the enterprise. With Keystone Juno, you can create a "users" domain with an LDAP identity backend for the enterprise user accounts, and use the "default" domain with the default sql identity backend. Note that Keystone Juno also recommends that the assignment information (roles, projects, domains) be kept separate from the identity information (users, groups), and recommends using the default sql assignment backend.

The following instructions assume that you already have a deployment of OpenStack Juno and want to add your enterprise LDAP as a new domain. These instructions also assume you are using IPA (Red Hat IdM) as your enterprise LDAP server with hostname, which stores user information in the suffix cn=accounts,dc=example,dc=com.

Keystone developer Adam Young has some similar instructions:

Scripted demo

There is a scripted demo which sets up two VMs: one running IPA, and one running OpenStack. See for more information.

Create an account for Keystone to use

Keystone must authenticate (e.g. LDAP BIND) to the enterprise LDAP server. You will need to create a user account with a password for Keystone to use. This user will need to have the right to perform searches and retrieve user information in the LDAP subtree which contains your user information (e.g. cn=users,cn=accounts,dc=example,dc=com). In addition, if you want Keystone to be able to write/update user information, you will have to grant the keystone user the right to add/update/delete user accounts. For this example, the user has the DN uid=keystone,cn=users,cn=accounts,dc=example,dc=com.

Create the new domain

The first step is to tell Keystone that you are using multiple domains. To do this, set the parameter domain_specific_drivers_enabled in the [identity] to the value True, and set the paramter domain_config_dir to the value /etc/keystone/domains. For example, using the command line tools as root:
## Enable domain specific config for Keystone
# openstack-config --set /etc/keystone/keystone.conf identity domain_specific_drivers_enabled true
# openstack-config --set /etc/keystone/keystone.conf identity domain_config_dir /etc/keystone/domains

The /etc/keystone/keystone.conf should look like this:
domain_specific_drivers_enabled = true
domain_config_dir = /etc/keystone/domains

Each domain is configured by a separate file in /etc/keystone/domains. If this directory does not exist, create it. Each domain is configured by a file called keystone.$domainname.conf in this directory, where $domainname is the name of your domain. The basic steps for configuring Keystone to use an LDAP identity backend are found here: and For example: to create a new domain called "users", with a read-only LDAP identity backend, create a file named /etc/keystone/domains/keystone.users.conf and populate it like this:

driver = keystone.identity.backends.ldap.Identity

Admin access and Policy

The use of domains requires the use of the Keystone v3 API, which in turn requires the use of Keystone v3 policy. In order to perform administrative actions over all domains, you must designate one of your domains as the "super" domain or admin domain. One or more users from this domain will be granted the right to administer all domains. It does not matter which domain it is.

First, use the current "admin" user to create the new "users" domain e.g. using the openstack command line tool. This assumes you have a user called "admin" in a project called "admin" both of which live in your default domain. Or you may have a file called /root/keystonerc_admin which sets up the environment for admin authentication, and you can grab the username, password, and tenant name. NOTE: with Keystone v3, "tenant" is now called "project".
openstack --os-identity-api-version 3 \
--os-auth-url \
--os-username admin \
--os-password myadminpassword \
--os-user-domain-name default \
--os-project-domain-name default \
--os-project-name admin \
--format shell \
domain create --description "Domain for enterprise users" --enable users

This will print out information about the new domain in the format keyword="value". Note the value of the id field. This is the domain id (without the double quotes).

Next, grab the default v3 policy template. This file is usually called /usr/share/keystone/policy.v3cloudsample.json. Copy this file to /etc/keystone. Edit /etc/keystone/policy.v3cloudsample.json - look for the string admin_domain_id near the top of the file. It should look like this before you edit it:
"admin_required": "role:admin",
"cloud_admin": "rule:admin_required and domain_id:admin_domain_id",
"service_role": "role:service",

You will need to replace admin_domain_id with the id of the domain you want to make your admin domain. If you want to use the default domain (where your OpenStack service accounts and probably your current admin user live), use a value of default. The file should look something like this:
"admin_required": "role:admin",
"cloud_admin": "rule:admin_required and domain_id:default",
"service_role": "role:service",

If you want to use your new users domain, use the domain id value from when you created that domain, which is usually a long base-16 string of hexadecimal digits, something like this:
"admin_required": "role:admin",
"cloud_admin": "rule:admin_required and domain_id:ad8d0d5fd7e84273a9c1024083743480",
"service_role": "role:service",

You will need to restart Keystone for the policy changes to take effect.

Role assignments

With Keystone v2, you could assign users to have roles within tenants. With Keystone v3, you can assign users to have roles within domains or projects (tenants are now called projects in Keystone v3). NOTE: In the examples below, the $admin_user and $admin_domain are the admin user and admin domain you configured in the steps above. If you want to assign roles to some of your users from your new "users" account, for example, to have the admin role in your domain:
openstack --os-identity-api-version 3 \
--os-auth-url \
--os-username $admin_user \
--os-password myadminpassword \
--os-user-domain-name $admin_domain \
--os-domain-name $admin_domain \
role add --domain users --user $id_of_user admin

The $id_of_user is used above. With Keystone v3, when referring to user and project resources, it is better to use the id rather than the name. You can have multiple users with the same name in different domains, but the id is guaranteed to be globally unique across all domains. When referring to domains, you can use either the id or the name, because there is only one namespace for domains. If you don't know the id, you can use openstack user list --long --domain domainname:
| ID | Name | Project Id | Domain Id | Description | Email | Enabled |
| 54ca2d88e1df460e824de237438a6af1 | swift | | default | | swift@localhost | True |
| 9706d7cea2b54692ae12269f662a5d13 | glance | | default | | glance@localhost | True |
| 9a3342d2b5ed4931a7a4f21d215f303d | admin | | default | | root@localhost | True |
| b9c40934c7df4208a4cf93d5205f18ba | cinder | | default | | cinder@localhost | True |
| c353ee3282a9400691665af6f6e412f0 | heat | | default | | heat@localhost | True |
| ca9bfb3ac14e469f9b3932af4395f825 | ceilometer | | default | | ceilometer@localhost | True |
| ceb8233429d44a1883c73184e825a214 | nova | | default | | nova@localhost | True |

or openstack user show username --domain domainname:
| Field | Value |
| default_project_id | 34f66a1a14694983838ee860b15bfb38 |
| domain_id | default |
| email | root@localhost |
| enabled | True |
| id | 9a3342d2b5ed4931a7a4f21d215f303d |
| name | admin |

To get the id for a project, use openstack project list --long or openstack project show projectname --domain domainname. These commands will give you the domain_id. If you want to see the domain name, use openstack domain list which will show the domain name and id.

To add a project to the new users domain e.g. called "demo":
openstack --os-identity-api-version 3 \
--os-auth-url \
--os-username $admin_user \
--os-password myadminpassword \
--os-user-domain-name $admin_domain \
--os-domain-name $admin_domain \
project create demo --domain users

To add a user to the new project and assign the user the role "_member_" in the project:
openstack --os-identity-api-version 3 \
--os-auth-url \
--os-username $admin_user \
--os-password myadminpassword \
--os-user-domain-name $admin_domain \
--os-domain-name $admin_domain \
role add --project $demo_project_id --user $id_of_user _member_

Dashboard/Horizon configuration

These are the steps to configure Dashboard/Horizon to use the new users domain for authentication. This will make it so that users in the new "users" domain can login via Dashboard/Horizon, but not users in other domains.
# cat >> /etc/openstack-dashboard/local_settings << EOF
"identity": 3

The file should look like this:
# For Glance image upload, Horizon uses the file upload support from Django
# so we add this option to change the directory where uploaded files are temporarily
# stored until they are loaded into Glance.
"identity": 3

This tells dashboard to use Keystone v3 for authentication and to use 'users' as the default domain.

More changes:
# sed -i "s/^OPENSTACK_KEYSTONE_URL = .*/OPENSTACK_KEYSTONE_URL = \"http:\/\/$VM_FQDN:5000\/v3\"/g" \
# sed -i "s/^ 'name': 'native',/ 'name': 'ldap',/g" \
# sed -i "s/^ 'can_edit_user': True,/ 'can_edit_user': False,/g" \
# sed -i "s/^ 'can_edit_group': True,/ 'can_edit_group': False,/g" \

The first line tells dashboard to use the Keystone v3 URL, that Keystone is using 'ldap' for auth, and that the user and group information cannot be edited via dashboard. The end result file should look like this:
# For multiple regions uncomment this configuration, and add (endpoint, title).

OPENSTACK_KEYSTONE_URL = "http://rdo.rdodom.test:5000/v3"
'name': 'ldap',
'can_edit_user': False,
'can_edit_group': False,
'can_edit_project': True,
'can_edit_domain': True,
'can_edit_role': True

Next, policy:
# mv /etc/openstack-dashboard/keystone_policy.json /etc/openstack-dashboard/keystone_policy.json.orig
# cp /etc/keystone/policy.v3cloudsample.json /etc/openstack-dashboard/keystone_policy.json

We need to tell Horizon/Dashboard that we are using the new v3 policy in Keystone. But first, make a copy of the original policy.

You will need to restart Horizon/Dashboard after making these changes.

Other OpenStack service configuration

Some of the other OpenStack services do not use Keystone v3 by default, or have bugs in that support. What follows are some configuration changes or workarounds:
## this is a workaround for LP#1427878 - LP is
## Update Nova to allow it to validate tokens using the
## v3 Identity API.
# sed -i "s/^auth_version\(.*\)/#auth_version\1/g" /usr/share/nova/nova-dist.conf

## this is a workaround for LP#1428376
## Update Swift to use the proper keystonemiddleware module. Without
## this, multi-domain support in Swift doesn't work properly.
# sed -i "s/^paste.filter_factory.*/paste.filter_factory = keystonemiddleware.auth_token:filter_factory/g" \

The /usr/share/nova/nova-dist.conf should look like this:
# Workaround for
#auth_version = v2.0

and the /etc/swift/proxy-server.conf should look like this:
log_name = swift
signing_dir = /var/cache/swift
paste.filter_factory = keystonemiddleware.auth_token:filter_factory

You will have to restart those services for the changes to take effect.

Create a v3 keystonerc file

Some installers will create a shell script rc file that you can source into the shell environment to make it easy to use the command line utilities. Here is an example of such a file to use the v3 api with the openstack command. NOTE: your_admin_user, your_admin_password, and your_admin_domain are the admin username, password, and admin domain you selected above:
export OS_USERNAME=your_admin_user
export OS_PASSWORD=your_admin_password
export OS_DOMAIN_NAME=your_admin_domain
export OS_USER_DOMAIN_NAME=your_admin_domain
export OS_AUTH_URL=
export PS1='[\u@\h \W(keystone_cloud_admin)]$ '

Specifying OS_IDENTITY_API_VERSION=3 and OS_AUTH_URL= tells openstack that you want to use the Keystone v3 API.

How to configure OpenStack Keystone to use LDAP - part 2 - foreman/staypuft


In Part 1 - packstack I talked about how to use packstack to deploy Keystone with an LDAP identity backend. In this part 2 I will describe how to do the same thing with a foreman/staypuft based installer. The basic steps for manually configuring Keystone to use an LDAP identity backend are found here: and

How to tell if my version of foreman/staypuft supports Keystone LDAP?

The puppet class quickstack::pacemaker::keystone has been extended with all of the Keystone LDAP parameters listed at Find your quickstack/manifests/pacemaker/keystone.pp file, and see if it has the keystone_identity_backend parameter along with many paramters of the form ldap_*. If so, then your foreman/staypuft has support for Keystone LDAP. It should look like this:
class quickstack::pacemaker::keystone (
$keystone_identity_backend = 'sql',
$ldap_url = '',
$ldap_user = '',


The first thing is to specify to use LDAP as the Keystone identity backend. The puppet class quickstack::pacemaker::keystone has a class parameter name keystone_identity_backend which takes the values sql (default) and ldap. Use "ldap" to use the Keystone LDAP identity backend. All of the configuration parameters listed here are supported: To specify one of those parameters puppet, just put "ldap_" in front of the name. For example: [ldap] url in puppet is ldap_url.

There are two ways to specify the configuration in foreman:

  • Using the UI: foreman allows you to set class parameters in the hostgroups UI. You would set the class parameters of the quickstack::pacemaker::keystone class.

  • Using the command line: All of the parameters can be set in the seeds.rb file. For example:
    params = {
    "verbose" => "true",
    "heat_cfn" => "true",
    "keystone_identity_backend" => "ldap",
    "ldap_url" => "ldap://myhostname:389/",
    ... other ldap parameters ...

At a minimum, you will need to specify the url, the user_dn, the password, the suffix, the user_tree_dn, and the group_tree_dn.

How to configure OpenStack Keystone to use LDAP - part 1 - packstack


The latest versions of packstack now have the ability to set up Keystone to use LDAP as its identity backend. The basic steps for manually configuring Keystone to use an LDAP identity backend are found here: and Packstack allows you to set up all of these parameters.

How to tell if my version of packstack supports Keystone LDAP?

packstack --help should list the option --keystone-identity-backend which takes the values sql (default value) and ldap. There should also be a large number of options in the form of --keystone-ldap-PARAM that allow you to set up all aspects of the Keystone LDAP identity backend.

Configuration with packstack

The first step is to tell packstack that you are using Keystone with an LDAP identity backend. Either use packstack --keystone-identity-backend ldap ...other options... or use CONFIG_KEYSTONE_IDENTITY_BACKEND=ldap in your packstack answer-file. packstack supports all of the configuration parameters listed here: To specify one of those values in packstack:

  • command line - add --keystone-ldap- in front of the parameter: [ldap] ldap_suffix becomes --keystone-ldap-suffix

  • answer-file - convert to all caps, and add CONFIG_KEYSTONE_LDAP_ in front of the parameter name: [ldap] ldap_suffix becomes CONFIG_KEYSTONE_LDAP_SUFFIX

NOTE: Two exceptions to the above rule

  • [ldap] user is --keystone-ldap-user-dn and CONFIG_KEYSTONE_LDAP_USER_DN in packstack

  • [ldap] password is --keystone-ldap-user-password and CONFIG_KEYSTONE_LDAP_USER_PASSWORD in packstack

At a minimum, you will need to specify the url, the user_dn, the password, the suffix, the user_tree_dn, and the group_tree_dn.

Running puppet-keystone spec tests in docker container

I am working on adding features to puppet-keystone such as support for configuring Keystone to use the LDAP identity backend, read-only as well as read-write, and to have multiple LDAP identity backends.  I wanted to easily run tests on different platforms using docker.

I started with a fedora 20 docker image, and installed the ruby and ruby bundle packages.  I called this image "puppet-keystone".  In order to share my development source code working directory, I mount my home directory inside the container as "/share".  In order for the container to have access, I need to create my user and group inside the container:

# groupadd -g 1000 rmeggins
# useradd -g rmeggins -u 1000 rmeggins

I ran docker like this:

$ sudo docker run -v /home/rmeggins:/share -i -t -u 1000 puppet-keystone

However, I could not see /share - permission denied.  Not sure why, but I figured that I had to run the container with --privileged like this:

$ sudo docker run --privileged -v /home/rmeggins:/share -i -t -u 1000 puppet-keystone

This allowed me full access to the /share volume.

Then I ran the test setup:

$ cd /share/puppet-keystone
$ GEM_HOME=vendor bundle install
$ GEM_HOME=vendor bundle exec rake spec SPEC_OPTS="--format d"

This caused errors reading from manifests/wsgi/apache.pp:

  1) keystone::wsgi::apache on RedHat platforms configures apache serving keysto
ne with mod_wsgi
     Failure/Error: it { should contain_service('httpd').with_name(platform_para
meters[:httpd_service_name]) }
       invalid byte sequence in US-ASCII at /share/puppet-keystone/spec/fixtures
/modules/keystone/manifests/wsgi/apache.pp:1 on node
     Shared Example Group: "apache serving keystone with mod_wsgi" called from .

Sure enough, file reports a different type for that file:

$ find manifests -name "*.pp" | xargs file
manifests/init.pp:                      ASCII text
manifests/endpoint.pp:                  C++ source, ASCII text
manifests/python.pp:                    C++ source, ASCII text
manifests/service.pp:                   C++ source, ASCII text
manifests/cron/token_flush.pp:          C++ source, ASCII text
manifests/wsgi/apache.pp:               C++ source, UTF-8 Unicode text

For some reason, inside the container, it can't read this file, but I don't have this problem from the host.  I was able to work around the problem in the container by running the test like this:

$ LC_CTYPE=C.UTF-8 LANG=C.UTF-8 GEM_HOME=vendor bundle exec rake spec SPEC_OPTS="--format d"

Now I am able to run the tests successfully with no errors