How to debug "undefined method for nil:NilClass" in OpenShift Aggregated Logging

In OpenShift Aggregated Logging the Fluentd pipeline tries very hard to ensure that the data is correct, because it depends on having clean data in the output section in order to construct the index names for Elasticsearch. If the fields and values are not correct, then the index name construction will fail with an unhelpful error like this:

2017-09-28 13:22:22 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-28 13:22:23 -0400 error_class="NoMethodError"
error="undefined method `[]' for nil:NilClass" plugin_id="object:1c0bd1c"
2017-09-28 13:22:22 -0400 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch- `eval'
2017-09-28 13:22:22 -0400 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch- `eval'

There is no context about what field might be missing, what tag is matching, or even which plugin it is, the operations output or the applications output (although you do get the plugin_id, which could be used to look up the actual plugin information, if the Fluentd monitoring is enabled).
One solution is to just edit the logging-fluentd ConfigMap, and add a stdout filter in the right place:
## matches
          <filter **>
            @type stdout
          @include configs.d/openshift/output-pre-*.conf

and dump the time, tag, and record just before the outputs. The problem with this is that it will cause a feedback loop, since Fluentd is reading from its own pod log. The solution to this is to also throw away Fluentd pod logs.
## filters
          @include configs.d/openshift/filter-pre-*.conf
          @include configs.d/openshift/filter-retag-journal.conf
          <match kubernetes.journal.container.fluentd kubernetes.var.log.containers.fluentd**>
            @type null

This must come after the filter-retag-journal.conf which identifies and tags Fluentd pod log records. Then restart Fluentd (oc pod delete $fluentd_pod, oc label node, etc.). The Fluentd pod log will now contain data like this:
2017-09-28 13:44:47 -0400 output_tag: {"type":"response","@timestamp":"2017-09-28T17:44:19.524989+00:00","pid":8,"method":"head","statusCode":200,
"message":"HEAD / 200 2ms - 9.0B",

Now, if you see a record that is missing @timestamp, or a record from a pod that is missing kubernetes.namespace_name or kubernetes.namespace_id, you know that the exception is caused by one of these missing fields.

Elasticsearch Troubleshooting - unassigned_shard and cluster state RED

Problem - unassigned_shards and cluster status RED

Using OpenShift origin-aggregated-logging 1.2, Elasticsearch 1.5.2, the cluster status is RED.
oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  https://localhost:9200/_cluster/health | \
  python -mjson.tool
    "active_primary_shards": 12345,
    "active_shards": 12345,
    "cluster_name": "logging-es",
    "initializing_shards": 0,
    "number_of_data_nodes": 3,
    "number_of_nodes": 3,
    "number_of_pending_tasks": 0,
    "relocating_shards": 0,
    "status": "red",
    "timed_out": false,
    "unassigned_shards": 7

The problem is the unassigned_shards. We need to identify those shards and
figure out how to deal with them so the cluster can move to yellow or

Solution - identify and delete problematic indices

Use the /_cluster/health?level=indices to get a list of the indices status:
oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  https://localhost:9200/_cluster/health?level=indices | \
  python -mjson.tool > indices.json

The report will list each index and its state:
   "active_primary_shards": 4,
   "active_shards": 4,
   "initializing_shards": 0,
   "number_of_replicas": 0,
   "number_of_shards": 5,
   "relocating_shards": 0,
   "status": "red",
   "unassigned_shards": 1

Look for records that have "status": "red" and an "unassigned_shards" with
THAT THIS DATA CAN BE LOST, then it might be easiest to just delete these using
oc exec logging-es-xxx-N-yyy -n logging -- curl -s \
  --key /etc/elasticsearch/keys/admin-key \
  --cert /etc/elasticsearch/keys/admin-cert \
  --cacert /etc/elasticsearch/keys/admin-ca \
  -XDELETE https://localhost:9200/my-index.2017.03.15

If you need to recover this data, or deletion is not working, then use the
recovery procedure documented at
indices recovery

Monitoring Fluentd and the Elasticsearch output plugin

Fluentd has a monitor input plugin:

Unfortunately, the documentation is pretty scant, and some of the useful, interesting endpoints and options are not documented. I've captured some of that missing information below, and shown how it can be used to monitor the Elasticsearch output plugin.



Provides information about each plugin in a text based columnar format:
$ curl -s http://localhost:24220/api/plugins
plugin_id:object:1dce4b0        plugin_category:input   type:monitor_agent
output_plugin:false     retry_count:
plugin_id:object:11b4120        plugin_category:input   type:systemd    output_p
lugin:false     retry_count:
plugin_id:object:19fb914        plugin_category:output  type:rewrite_tag_filter
output_plugin:true      retry_count:


Same as /api/plugins except in JSON format:
$ curl -s http://localhost:24220/api/plugins.json | python -mjson.tool
    "plugins": [
            "config": {
                "@type": "monitor_agent",
                "bind": "",
                "port": "24220"
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"


Provides basic fluentd configuration information in text format:
$ curl -s http://localhost:24220/api/config
pid:19  ppid:1  config_path:/etc/fluent/fluent.conf     pid_file:       plugin_dirs:["/etc/fluent/plugin"]      log_path:


Provides basic fluentd configuration information in JSON format:
$ curl -s http://localhost:24220/api/config.json | python -mjson.tool
    "config_path": "/etc/fluent/fluent.conf",
    "log_path": null,
    "pid": 19,
    "pid_file": null,
    "plugin_dirs": [
    "ppid": 1

Query String Options


For plugins, this will print all of the instance variables:
$ http://localhost:24220/api/plugins.json\?debug=1 | python -mjson.tool
    "plugins": [
            "config": {
                "@type": "monitor_agent",
                "bind": "",
                "port": "24220"
            "instance_variables": {
                "bind": "",
                "emit_config": false,
                "emit_interval": 60,


Search for plugin by @type:
$ http://localhost:24220/api/plugins.json\?@type=monitor_agent | python -mjson.tool
    "plugins": [
            "config": {
                "@type": "monitor_agent",
                "bind": "",
                "port": "24220"
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"


Search for plugin by @id. For example, in the above output, there is "plugin_id": "object:1dce4b0". Once you have identified the id, you can use that to display only the information for that particular id:
$ http://localhost:24220/api/plugins.json\?@id=object:1dce4b0 | python -mjson.tool
    "plugins": [
            "config": {
                "@type": "monitor_agent",
                "bind": "",
                "port": "24220"
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"


Match the tag and get the info from the matched output plugin. Only works on output plugins. I unfortunately don't have an example, but I suppose you could use something like this to find the output plugins which have a match block which has a match for **_sendtoforwarder_**:
$ http://localhost:24220/api/plugins.json\?tag=prefix_sendtoforwarder_suffix | python -mjson.tool
    "plugins": [

Debugging the Fluentd Elasticsearch plugin

First, identify the output plugin in question to get the plugin id:
$ http://localhost:24220/api/plugins.json\?@type=elasticsearch_dynamic | python -mjson.tool
    "plugins": [
            "buffer_queue_length": 0,
            "buffer_total_queued_size": 0,
            "config": {
                "@type": "elasticsearch_dynamic",
                "index_name": ".operations.${record['@timestamp'].nil? ?
(time).getutc.strftime(@logstash_dateformat) : Time.parse(record['@timestamp']).
            "plugin_id": "object:1b4cc64",

This is the one I'm looking for, which has a plugin id of object:1b4cc64. Next, I can use the @id parameter in conjunction with the debug one to get some interesting statistics:
$ http://localhost:24220/api/plugins.json\?@id=object:1b4cc64\&debug=1 | \
  python -mjson.tool | \
  egrep 'buffer_total_queued_size|emit_count'
            "buffer_total_queued_size": 0,
                "emit_count": 3164,

I can even put this in a simple loop to see how the queue size and emit count change over time:
$ while true ; do
  http://localhost:24220/api/plugins.json\?@id=object:1b4cc64\&debug=1 | \
    python -mjson.tool | egrep 'buffer_total_queued_size|emit_count'
  sleep 1
Wed Dec  7 23:56:18 UTC 2016
            "buffer_total_queued_size": 0,
                "emit_count": 3318,
Wed Dec  7 23:56:21 UTC 2016
            "buffer_total_queued_size": 1654,
                "emit_count": 3322,
Wed Dec  7 23:56:23 UTC 2016
            "buffer_total_queued_size": 2146,
                "emit_count": 3324,
Wed Dec  7 23:56:25 UTC 2016
            "buffer_total_queued_size": 0,
                "emit_count": 3326,

This tells me that the plugin is working, the queues are being flushed regularly, and the emit count (roughly, the number of times fluentd flushes the queued outputs, the number of times a request is made to Elasticsearch) is steadily increasing.

External Elasticsearch route with OpenShift logging


This has been fixed in openshift-elasticseach-plugin, which is used with 1.4.1/3.4.1 version of OpenShift. If you are using an earlier version, the warning below applies.

There is no actual authentication check performed on each request. Do not use the below if you need actual security and authentication - for throwaway dev environments only. If you need a secure method, you must instead use a passthrough route, and use mutual (i.e. client cert) authentication.


The Elasticsearch deployed with OpenShift aggregated logging is not accessible externally, outside the logging cluster, by default. The intention is that Kibana will be used to access the data, and the various ways to deploy/install OpenShift with logging allow you to specify the externally visible hostname that Kibana (including the separate operations cluster) will use. However, there are many tools that want to access the data from Elasticsearch. This post describes how to enable a route for external access to Elasticsearch.

You will first need an FQDN for the Elasticsearch (and a separate FQDN for the Elasticsearch ops instance if using the separate operations cluster). I am testing with an all-in-one (OpenShift master + node + logging components) install on an OpenStack machine, which has a private IP and hostname, and a public (floating) IP and hostname. In a real deployment, the public IP addresses and hostnames for the elasticsearch services will need to be added to DNS.
private host, IP: host-192-168-78-2.openstacklocal,
public host, IP: run-logging-source.oshift.rmeggins.test.novalocal, 10.x.y.z 

I have done the following on my local machine and in the all-in-one machine, by hacking /etc/hosts. All-in-one machine:   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.x.y.z run-logging-source.oshift.rmeggins.test.novalocal

My local machine:
10.x.y.z run-logging-source.oshift.rmeggins.test.novalocal run-logging-source.oshift.rmeggins.test

I set up a router after installing OpenShift:
$ oc create serviceaccount router -n default
$ oadm policy add-scc-to-user privileged system:serviceaccount:default:router
$ oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:default:router
$ oadm router --create --namespace default --service-account=router \
     --credentials $MASTER_CONFIG_DIR/openshift-router.kubeconfig

$ oc get pods -n default
NAME                      READY     STATUS    RESTARTS   AGE
docker-registry-1-7z0gq   1/1       Running   0          35m
router-1-8bp88            1/1       Running   0          24m

$ oc logs -n default router-1-8bp88
I1010 19:57:57.815578       1 router.go:161] Router is including routes in all namespaces
I1010 19:57:57.922277       1 router.go:404] Router reloaded:
 - Checking HAProxy /healthz on port 1936 ...
 - HAProxy port 1936 health check ok : 0 retry attempt(s).

Logging setup should have already created services for Elasticsearch:
$ oc project logging
$ oc get svc
NAME                     CLUSTER-IP       EXTERNAL-IP   PORT(S)  AGE
logging-es         none          9200/TCP 33m
logging-es-ops    none          9200/TCP 33m

The route is a reencrypt route. The --dest-ca-cert argument value below is the CA cert for the CA that issued the Elasticsearch server cert, used to re-encrypt the connection from the router to Elasticsearch. In this case, it is the same as the admin-ca cert, so we can just use that (using the method to extract it from the previous posting). By default, the route will use the server cert created by the OpenShift master CA. If you want to have a real server cert with the actual external Elasticsearch hostname, you will need to create one. An example of how to do this with OpenShift is described below (marked #optional). The route allows us to use username/password/token authentication to Elasticsearch - the auth is proxied through the router to SearchGuard/Elasticsearch.
$ ca=`mktemp`
$ cert=`mktemp`
$ key=`mktemp`
$ oc get secret logging-elasticsearch \
    --template='{{index .data "admin-ca"}}' | base64 -d > $ca
#optional - MASTER_CONFIG_DIR e.g. /etc/origin/master
$ openshift admin ca create-server-cert --key=es.key \
          --cert=es.crt --hostnames=es.fqdn.hostname \
          --signer-cert=$MASTER_CONFIG_DIR/ca.crt \
          --signer-key=$MASTER_CONFIG_DIR/ca.key \
#optional - MASTER_CONFIG_DIR e.g. /etc/origin/master
$ openshift admin ca create-server-cert --key=es-ops.key \
          --cert=es-ops.crt --hostnames=es-ops.fqdn.hostname \
          --signer-cert=$MASTER_CONFIG_DIR/ca.crt \
          --signer-key=$MASTER_CONFIG_DIR/ca.key \
$ oc create route -n logging reencrypt --service logging-es \
                        --port 9200 --hostname es.fqdn.hostname \
                        --dest-ca-cert=$ca \
                        #optional --ca-cert=$MASTER_CONFIG_DIR/ca.crt --cert=es.crt --key=es.key
$ oc create route -n logging reencrypt --service logging-es-ops \
                         --port 9200 --hostname es-ops.fqdn.hostname \
                         --dest-ca-cert=$ca \
                         #optional --ca-cert=$MASTER_CONFIG_DIR/ca.crt --cert=es-ops.crt --key=es-ops.key

I'm using the AllowAll identity provider so I can just create users/passwords with oc login (for testing):
$ more /tmp/openshift/origin-aggregated-logging/openshift.local.config/master/master-config.yaml
  - challenge: true
    login: true
    mappingMethod: claim
    name: anypassword
      apiVersion: v1
      kind: AllowAllPasswordIdentityProvider

I create a user called "kibtest" (I also use this user for kibana testing) that has cluster admin rights:
$ oc login --username=system:admin
$ oc login --username=kibtest --password=kibtest
$ oc login --username=system:admin
$ oadm policy add-cluster-role-to-user cluster-admin kibtest

I get the username and token for kibtest:
$ oc login --username=kibtest --password=kibtest
$ test_token="$(oc whoami -t)"
$ test_name="$(oc whoami)"
$ test_ip=""
$ oc login --username=system:admin

Now I can use curl like this:
$ curl -s -k -H "X-Proxy-Remote-User: $test_name" -H "Authorization: Bearer $test_token" -H "X-Forwarded-For:" https://es.fqdn.hostname
  "name" : "Sugar Man",
  "cluster_name" : "logging-es",
  "version" : {
    "number" : "2.3.5",
    "build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
    "build_timestamp" : "2016-07-27T10:36:52Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  "tagline" : "You Know, for Search"

$ curl -s -k -H "X-Proxy-Remote-User: $test_name" -H "Authorization: Bearer $test_token" -H "X-Forwarded-For:" https://es-ops.fqdn.hostname/.operations.*/_search?q=message:centos | python -mjson.tool | more
    "_shards": {
        "failed": 0,
        "successful": 1,
        "total": 1
    "hits": {
        "hits": [
                "_id": "AVewK5inAJ6n02oOdaIc",
                "_index": ".operations.2016.10.10",
                "_score": 11.1106205,
                "_source": {
                    "@timestamp": "2016-10-10T19:46:43.000000+00:00",
                    "hostname": "host-192-168-78-2.openstacklocal",
                    "ident": "docker-current",
                    "ipaddr4": "",
                    "ipaddr6": "fe80::42:acff:fe11:5",
                    "message": "time=\"2016-10-10T19:46:43.564686094Z\" level=in....."

Works the same from my local machine.

How to print field name with dash ("-") in a golang template

For example, let's say your OpenShift secret has been created like this:
$ oc secrets new logging-elasticsearch \
        key=$dir/keystore.jks truststore=$dir/truststore.jks \
        searchguard.key=$dir/searchguard_node_key \
        searchguard.truststore=$dir/searchguard_node_truststore \
        admin-key=$dir/${admin_user}.key admin-cert=$dir/${admin_user}.crt \
        admin-ca=$dir/ca.crt \

Now you want to extract the CA cert:
$ oc get secret logging-elasticsearch --template='{{.data.admin-ca}}'
error: error parsing template {{.data.admin-ca}}, template: output:1: bad character U+002D '-'

It doesn't like the - character in the field name. You can work around this using index like so:
$ oc get secret logging-elasticsearch --template='{{index .data "admin-ca"}}' |base64 -d > ca
$ openssl x509 -in ca -text|more
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=logging-signer-20160915173520
            Not Before: Sep 15 17:35:19 2016 GMT
            Not After : Sep 14 17:35:20 2021 GMT
        Subject: CN=logging-signer-20160915173520
        Subject Public Key Info:

How to do python dict setdefault with ruby hashes

setdefault is a very useful Python Dict method.
Python 2.7.11 (default, Jul  8 2016, 19:45:00) 
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> dd = {}
>>> dd.setdefault('a', {}).setdefault('b', {})['c'] = 'd'
>>> dd
{'a': {'b': {'c': 'd'}}}
>>> dd.setdefault('a', {}).setdefault('b', {})['e'] = 'f'
>>> dd
{'a': {'b': {'c': 'd', 'e': 'f'}}}
>>> dd.setdefault('g', {}).setdefault('b', {})['e'] = 'f'
>>> dd
{'a': {'b': {'c': 'd', 'e': 'f'}}, 'g': {'b': {'e': 'f'}}}

You can do the same thing in ruby with a little hackery.
irb(main):001:0> dd = {}
=> {}
irb(main):002:0> ((dd['a'] ||= {})['b'] ||= {})['c'] = 'd'
=> "d"
irb(main):003:0> dd
=> {"a"=>{"b"=>{"c"=>"d"}}}
irb(main):004:0> ((dd['a'] ||= {})['b'] ||= {})['e'] = 'f'
=> "f"
irb(main):005:0> dd
=> {"a"=>{"b"=>{"c"=>"d", "e"=>"f"}}}
irb(main):006:0> ((dd['g'] ||= {})['b'] ||= {})['e'] = 'f'
=> "f"
irb(main):007:0> dd
=> {"a"=>{"b"=>{"c"=>"d", "e"=>"f"}}, "g"=>{"b"=>{"e"=>"f"}}}

How to find build-time vs. run-time dependencies of a gem

Using ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-linux]
gem2rpm 0.11.3
gem 2.4.8

I'm trying to convert gems to rpms. Unfortunately, gem2rpm -d does not separate/classify the dependencies. What I really need is a separate list of run-time dependencies. I can get this with gem spec --ruby. For example:
$ gem spec --ruby systemd-journal-1.2.2.gem
# -*- encoding: utf-8 -*-
# stub: systemd-journal 1.2.2 ruby lib do |s| = "systemd-journal"
  s.version = "1.2.2"
  if s.respond_to? :specification_version then
    s.specification_version = 4

    if >='1.2.0') then
      s.add_runtime_dependency(%q<ffi>, ["~> 1.9.0"])
      s.add_development_dependency(%q<rspec>, ["~> 3.1"])
      s.add_development_dependency(%q<simplecov>, ["~> 0.9"])
      s.add_development_dependency(%q<rubocop>, ["~> 0.26"])
      s.add_development_dependency(%q<rake>, ["~> 10.3"])
      s.add_development_dependency(%q<yard>, ["~> 0.8.7"])
      s.add_development_dependency(%q<pry>, ["~> 0.10"])

So I need to add Requires: rubygem(ffi) to the spec.

How to fix docker when it cannot pull due to "x509: certificate signed by unknown authority"

I've been having this problem on Fedora 23 with docker 1.9.1 build ee06d03/1.9.1.  When I would use docker pull, it would give me a cert error:
 # docker pull some/image:tag
 Trying to pull repository ... failed
 Error while pulling image: Get x509: certificate signed by unknown authority
Not sure why docker can't just use the system cert bundle. Looking at the code: docker looks for /etc/docker/certs.d/$hostname and looks for a CA cert bundle in that directory. So I just did this:
 # cd /etc/docker/certs.d
 ln -s /etc/pki/tls/certs/ca-bundle.crt
 ln -s /etc/pki/tls/certs/
 systemctl restart docker
Now docker pull works fine for the Dockerhub repo.

ViaQ - data (log, event, telemetry) aggregation, correlation, viewing, analysis

ViaQ -

Modern environments become more and more complex every year.  When many applications and services collaborate together to perform a single task finding a cause of a problem is similar to looking for a needle in a haystack. Good tools are needed to help. There are some that do a very good job of collecting logs, alerts or notifications but they focus on a specific problem and not on the problem space as a whole. Collecting just logs, alerts or statistical data is not enough. There needs to be a way to combine the data together and let it speak, so that data from many different applications can be correlated from end-to-end, and from high to low levels.  ViaQ is a new project that aims at creating a framework for connecting data aggregation, processing, and analytic technologies that already exist into a coherent and flexible solution adaptable to multiple use cases.

There are some efforts that we want to leverage:

  • OpenShift has begun shipping an EFK stack as containers - we want to leverage this work to provide our solution as containers, but perhaps not dependent on OpenShift

  • There has been a lot of investigation of collecting event data such as logs using a message bus and feeding that data into analysis tools such as Apache Storm and Apache Spark - we would like to use a message bus based approach so that we can not only feed data to an EFK stack but at the same time feed data to an analytics tool, data warehouses, or any other application requiring a live stream of data

  • There has been a lot of work done to describe a common data format so that logs from OpenStack (all of the various components and log formats if different from oslo logging), Ceph/Gluster, and syslog can be correlated together (e.g. timestamps, hostnames, node identifiers, etc.)

  • Use the new CentOS infrastructure to build upstream images based on CentOS, use the CentOS CI, and eventually use the CentOS container image build and repository systems

Please check out for an example application using atomic, or for a shell script which uses just plain Docker containers.

Using RHEL Identity Management to automatically join VMs created with OSP7 (OpenStack) Nova


This is a demonstration about how to use RHEL Identity Management to automatically join VMs created with OSP7 (OpenStack) Nova, to automatically assign new VMs to hostgroups, and to automatically create DNS records when a floating IP address is assigned to a VM.

NOTE: The demo shows the ipaotp in the server instance metadata.  The latest code at uses the inject_files method to inject a file into the new VM containing the OTP, which means the OTP is not available to be queried, and the VM can erase it as soon as possible.

How it works

OpenStack Nova provides hooks which allow developers to create custom code using the internal Nova APIs to perform actions based on Nova actions.  The demonstration makes use of the build_instance and the instance_network_info hooks.  Here is the source of the hook implementation:  The build_instance.pre hook calls Identity Management with the host-add command.  This will essentially "reserve" a slot for the new host, but the new host will not be fully joined (i.e. able to use Kerberos, SSH, SSSD, etc.) until the ipa-client-install completes. The build_instance.pre hook then creates the parameters that it needs to specify as arguments for the host-add command.  It generates a One Time Password (OTP), and stores the OTP as a file named "/tmp/ipaotp" in the list of injected files in the new VM.  This allows the VM to specify the OTP as the -w argument of ipa-client-install, then delete the OTP after it has been used.  The OTP is used as the userpassword parameter for the host-add call. The ipaclass metadata item was set by using the --property argument with openstack server create.  The value of that item is set to be the value of the userclass parameter for the host-add call, which in the demo is used to automatically assign the new VM to a hostgroup. The fully qualified hostname is constructed by using the VM name as the leftmost component of the FQDN, and the domain used is the Nova dhcp_domain setting if available, or an IPA specific domain configuration parameter.  The force parameter is set to True because we want host-add to add the host even though we don't have a "real" public IP address yet, only the private IP address assigned by OpenStack networking.  The other parameters are provided to show what options are available when calling host-add.

The VM image provided in the demo uses cloud-init, and Nova has been set up to provide certain data for the VM to use with cloud-init to call ipa-client-install with the OTP.  The demo sets up the Nova vendordata_jsonfile_path with a JSON file containing the list of Identity Management client packages to install in the VM, and a runcmd to run a shell script that will run ipa-client-install.  The build_instance.pre hook has been configured to add that shell script in the list of injected files in the new VM.  The shell script extracts the OTP from /tmp/ipaotp, erases the file, then runs ipa-client-install -w $ipaotp -U.  Once this command completes successfully, the VM is fully joined to Identity Management, and users can SSH into the new machine.

The is called after Nova handles networking related events.  If the hook detects that there is a floating IP assignment, it calls dnsrecord-add to add the record for the floating IP address to the host in Identity Management.


The hook uses a file called /etc/nova/ipaclient.conf to store its configuration.  It requires the following configuration parameters:

  • service_name - The name of the Kerberos principal of the Identity Management HTTP JSON API service

  • url - The URL of the Identity Management HTTP JSON API service

  • cacert - The name of the file containing the certificate of the CA of the Identity Management HTTP JSON API service

  • keytab - The hook requires a user account in Identity Management that has the ability to add hosts and create DNS records.  The hook must be provided with a keytab file for this user.

  • connect_retries - How many times the hook will retry an API call

  • json_rpc_version - The version of the Identity Management HTTP JSON API that the hook is using

  • inject_files - Files to inject into the VM. The format is "/localpath/to/file[ /path/to/file/invm]".  If /path/to/file/invm is not given, then the path in the VM is assumed to be the same as the path in the local machine.