Log in

For example, let's say your OpenShift secret has been created like this:
$ oc secrets new logging-elasticsearch \
        key=$dir/keystore.jks truststore=$dir/truststore.jks \
        searchguard.key=$dir/searchguard_node_key \
        searchguard.truststore=$dir/searchguard_node_truststore \
        admin-key=$dir/${admin_user}.key admin-cert=$dir/${admin_user}.crt \
        admin-ca=$dir/ca.crt \

Now you want to extract the CA cert:
$ oc get secret logging-elasticsearch --template='{{.data.admin-ca}}'
error: error parsing template {{.data.admin-ca}}, template: output:1: bad character U+002D '-'

It doesn't like the - character in the field name. You can work around this using index like so:
$ oc get secret logging-elasticsearch --template='{{index .data "admin-ca"}}' |base64 -d > ca
$ openssl x509 -in ca -text|more
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=logging-signer-20160915173520
            Not Before: Sep 15 17:35:19 2016 GMT
            Not After : Sep 14 17:35:20 2021 GMT
        Subject: CN=logging-signer-20160915173520
        Subject Public Key Info:
setdefault is a very useful Python Dict method.
Python 2.7.11 (default, Jul  8 2016, 19:45:00) 
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> dd = {}
>>> dd.setdefault('a', {}).setdefault('b', {})['c'] = 'd'
>>> dd
{'a': {'b': {'c': 'd'}}}
>>> dd.setdefault('a', {}).setdefault('b', {})['e'] = 'f'
>>> dd
{'a': {'b': {'c': 'd', 'e': 'f'}}}
>>> dd.setdefault('g', {}).setdefault('b', {})['e'] = 'f'
>>> dd
{'a': {'b': {'c': 'd', 'e': 'f'}}, 'g': {'b': {'e': 'f'}}}

You can do the same thing in ruby with a little hackery.
irb(main):001:0> dd = {}
=> {}
irb(main):002:0> ((dd['a'] ||= {})['b'] ||= {})['c'] = 'd'
=> "d"
irb(main):003:0> dd
=> {"a"=>{"b"=>{"c"=>"d"}}}
irb(main):004:0> ((dd['a'] ||= {})['b'] ||= {})['e'] = 'f'
=> "f"
irb(main):005:0> dd
=> {"a"=>{"b"=>{"c"=>"d", "e"=>"f"}}}
irb(main):006:0> ((dd['g'] ||= {})['b'] ||= {})['e'] = 'f'
=> "f"
irb(main):007:0> dd
=> {"a"=>{"b"=>{"c"=>"d", "e"=>"f"}}, "g"=>{"b"=>{"e"=>"f"}}}
Using ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-linux]
gem2rpm 0.11.3
gem 2.4.8

I'm trying to convert gems to rpms. Unfortunately, gem2rpm -d does not separate/classify the dependencies. What I really need is a separate list of run-time dependencies. I can get this with gem spec --ruby. For example:
$ gem spec --ruby systemd-journal-1.2.2.gem
# -*- encoding: utf-8 -*-
# stub: systemd-journal 1.2.2 ruby lib

Gem::Specification.new do |s|
  s.name = "systemd-journal"
  s.version = "1.2.2"
  if s.respond_to? :specification_version then
    s.specification_version = 4

    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
      s.add_runtime_dependency(%q<ffi>, ["~> 1.9.0"])
      s.add_development_dependency(%q<rspec>, ["~> 3.1"])
      s.add_development_dependency(%q<simplecov>, ["~> 0.9"])
      s.add_development_dependency(%q<rubocop>, ["~> 0.26"])
      s.add_development_dependency(%q<rake>, ["~> 10.3"])
      s.add_development_dependency(%q<yard>, ["~> 0.8.7"])
      s.add_development_dependency(%q<pry>, ["~> 0.10"])

So I need to add Requires: rubygem(ffi) to the spec.


I've been having this problem on Fedora 23 with docker 1.9.1 build ee06d03/1.9.1.  When I would use docker pull, it would give me a cert error:
 # docker pull some/image:tag
 Trying to pull repository docker.io/some/image ... failed
 Error while pulling image: Get https://index.docker.io/v1/repositories/some/image/images: x509: certificate signed by unknown authority
Not sure why docker can't just use the system cert bundle. Looking at the code: https://github.com/docker/docker/blob/1061c56a5fc126a76344ea9dca9aa5f5e75eb902/registry/registry.go#L102 docker looks for /etc/docker/certs.d/$hostname and looks for a CA cert bundle in that directory. So I just did this:
 # cd /etc/docker/certs.d
 mkdir docker.io
 cd docker.io
 ln -s /etc/pki/tls/certs/ca-bundle.crt
 ln -s /etc/pki/tls/certs/ca-bundle.trust.crt
 systemctl restart docker
Now docker pull works fine for the Dockerhub repo.
ViaQ - https://github.com/ViaQ

Modern environments become more and more complex every year.  When many applications and services collaborate together to perform a single task finding a cause of a problem is similar to looking for a needle in a haystack. Good tools are needed to help. There are some that do a very good job of collecting logs, alerts or notifications but they focus on a specific problem and not on the problem space as a whole. Collecting just logs, alerts or statistical data is not enough. There needs to be a way to combine the data together and let it speak, so that data from many different applications can be correlated from end-to-end, and from high to low levels.  ViaQ is a new project that aims at creating a framework for connecting data aggregation, processing, and analytic technologies that already exist into a coherent and flexible solution adaptable to multiple use cases.

There are some efforts that we want to leverage:

  • OpenShift has begun shipping an EFK stack as containers - we want to leverage this work to provide our solution as containers, but perhaps not dependent on OpenShift

  • There has been a lot of investigation of collecting event data such as logs using a message bus and feeding that data into analysis tools such as Apache Storm and Apache Spark - we would like to use a message bus based approach so that we can not only feed data to an EFK stack but at the same time feed data to an analytics tool, data warehouses, or any other application requiring a live stream of data

  • There has been a lot of work done to describe a common data format so that logs from OpenStack (all of the various components and log formats if different from oslo logging), Ceph/Gluster, and syslog can be correlated together (e.g. timestamps, hostnames, node identifiers, etc.)

  • Use the new CentOS infrastructure to build upstream images based on CentOS, use the CentOS CI, and eventually use the CentOS container image build and repository systems

Please check out https://github.com/ViaQ/efk-atomicapp for an example application using atomic, or https://github.com/ViaQ/integration-tests for a simple-test.sh shell script which uses just plain Docker containers.


This is a demonstration about how to use RHEL Identity Management to automatically join VMs created with OSP7 (OpenStack) Nova, to automatically assign new VMs to hostgroups, and to automatically create DNS records when a floating IP address is assigned to a VM.

NOTE: The demo shows the ipaotp in the server instance metadata.  The latest code at https://github.com/richm/rdo-vm-factory/blob/master/rdo-ipa-nova uses the inject_files method to inject a file into the new VM containing the OTP, which means the OTP is not available to be queried, and the VM can erase it as soon as possible.

How it works

OpenStack Nova provides hooks http://docs.openstack.org/developer/nova/hooks.html which allow developers to create custom code using the internal Nova APIs to perform actions based on Nova actions.  The demonstration makes use of the build_instance and the instance_network_info hooks.  Here is the source of the hook implementation: https://github.com/richm/rdo-vm-factory/blob/master/rdo-ipa-nova/novahooks.py.  The build_instance.pre hook calls Identity Management with the host-add command.  This will essentially "reserve" a slot for the new host, but the new host will not be fully joined (i.e. able to use Kerberos, SSH, SSSD, etc.) until the ipa-client-install completes. The build_instance.pre hook then creates the parameters that it needs to specify as arguments for the host-add command.  It generates a One Time Password (OTP), and stores the OTP as a file named "/tmp/ipaotp" in the list of injected files in the new VM.  This allows the VM to specify the OTP as the -w argument of ipa-client-install, then delete the OTP after it has been used.  The OTP is used as the userpassword parameter for the host-add call. The ipaclass metadata item was set by using the --property argument with openstack server create.  The value of that item is set to be the value of the userclass parameter for the host-add call, which in the demo is used to automatically assign the new VM to a hostgroup. The fully qualified hostname is constructed by using the VM name as the leftmost component of the FQDN, and the domain used is the Nova dhcp_domain setting if available, or an IPA specific domain configuration parameter.  The force parameter is set to True because we want host-add to add the host even though we don't have a "real" public IP address yet, only the private IP address assigned by OpenStack networking.  The other parameters are provided to show what options are available when calling host-add.

The VM image provided in the demo uses cloud-init, and Nova has been set up to provide certain data for the VM to use with cloud-init to call ipa-client-install with the OTP.  The demo sets up the Nova vendordata_jsonfile_path with a JSON file containing the list of Identity Management client packages to install in the VM, and a runcmd to run a shell script that will run ipa-client-install.  The build_instance.pre hook has been configured to add that shell script in the list of injected files in the new VM.  The shell script extracts the OTP from /tmp/ipaotp, erases the file, then runs ipa-client-install -w $ipaotp -U.  Once this command completes successfully, the VM is fully joined to Identity Management, and users can SSH into the new machine.

The instance_network_info.post is called after Nova handles networking related events.  If the hook detects that there is a floating IP assignment, it calls dnsrecord-add to add the record for the floating IP address to the host in Identity Management.


The hook uses a file called /etc/nova/ipaclient.conf to store its configuration.  It requires the following configuration parameters:

  • service_name - The name of the Kerberos principal of the Identity Management HTTP JSON API service

  • url - The URL of the Identity Management HTTP JSON API service

  • cacert - The name of the file containing the certificate of the CA of the Identity Management HTTP JSON API service

  • keytab - The hook requires a user account in Identity Management that has the ability to add hosts and create DNS records.  The hook must be provided with a keytab file for this user.

  • connect_retries - How many times the hook will retry an API call

  • json_rpc_version - The version of the Identity Management HTTP JSON API that the hook is using

  • inject_files - Files to inject into the VM. The format is "/localpath/to/file[ /path/to/file/invm]".  If /path/to/file/invm is not given, then the path in the VM is assumed to be the same as the path in the local machine.

New project: Nunc Stans

The poll() based event framework for 389 is being revamped to use a new framework based on the project Nunc Stans.  Nunc Stans is a thread pool based event framework wrapper.  That is, it provides a layer on top of event frameworks like libevent, tevent, libev, etc., or directly on top of epoll() (if we didn't want to use an event framework).  Nunc Stans has some features that are interesting to 389:

  • It provides thread safe access to the event framework, allowing the use of non-thread safe event frameworks.

  • It provides timed I/O events to facilitate idle timeouts.

  • It provides the ability to run event callbacks in a thread pool or in the main event loop thread.

  • It uses lock free data structures to reduce thread contention.

The project is hosted at http://fedorahosted.org/nunc-stans
Information about how 389 plans to use nunc-stans is here http://www.port389.org/docs/389ds/design/nunc-stans.html


By default, the Red Hat/Fedora/RDO openstack-keystone package provides a systemd service file openstack-keystone.service for managing the eventlet-based Keystone daemons. If you configure Keystone to use HTTPD via Apache modwsgi, the openstack-keystone.service no longer works, and you get strange errors if you try to use it:
 # systemctl status openstack-keystone
 Failed: error ....
 # systemctl start openstack-keystone
 Failed: port is in use: 35357
 Failed: port is in use: 5000
 ... other errors ...

See https://bugzilla.redhat.com/show_bug.cgi?id=1213149 for more details.

The service doesn't know that Keystone is no longer a standalone service, but a webapp controlled by httpd. You can use this "trick" to "alias" openstack-keystone to httpd:
 # ln -s /usr/lib/systemd/system/httpd.service /etc/systemd/system/openstack-keystone.service
 # systemctl daemon-reload

This will override the /usr/lib/systemd/system/openstack-keystone.service provided with the openstack-keystone package. Now, when you execute a command related to openstack-keystone, it will instead be redirected to httpd, and you will see the status of httpd instead.
Puppet is a bit of a stickler about making sure class parameter => value statements line up correctly like this:
  my_resource { 'resource_name':
   short_name            => 'value',
   a_very_very_long_name => 'value',
   another_name          => 'value,

It can become very tiresome when adding a new parameter inside a long list of class parameters to have to go back and line up all of those =>. Fortunately, I use Emacs to edit Puppet code, so I can use the align-regexp function:

  • Highlight the region you want to align
  • Type M-x align-regexp
  • Type =>

et voila! Your class parameter assignments are nicely aligned. Thanks Emacs!


Monitoring epoll events

When doing analysis of daemon processes that use epoll, you may want to take a peek inside the epoll structure. This is available via the /proc/$PID/fdinfo.

  1. Find the process id (PID) of the daemon process e.g. # pidof procname

  2. Find the epoll file descriptor (fd) e.g. # epollfd=`ls -al /proc/$PID/fd | awk '/eventpoll/ {print $9}'`

  3. Dump the information about the epoll fd e.g. # cat /proc/$PID/fdinfo/$epollfd

The information looks like this:
pos:    0
flags:  02000002
tfd:        6 events:       19 data:                6
tfd:       20 events:       19 data:               14
tfd:       21 events:       19 data:               15
tfd:     2446 events:       19 data:              98e
tfd:     1991 events:       19 data:              7c7
tfd:     1065 events:       19 data:              429
tfd:     2444 events:       19 data:              98c

Each tfd is the decimal number of the file descriptor being polled by epoll. The events field is the hex value of the flags specified in the epoll_event.events field when calling epoll_ctl. 0x19 is EPOLLIN|EPOLLERR|EPOLLHUP. The data field is the hex value of the tfd decimal value.

If you suspect that your application is leaking file descriptors, and you believe the problem is that the application is adding fds to epoll but never deleting them, you can periodically poll the epoll fdinfo, to look for descriptors that are persistent. You should first figure out which file descriptors your application uses as persistent descriptors e.g. the listener sockets or the eventfd notifier.