?

Log in

No account? Create an account

Previous Entry | Next Entry

Fluentd has a monitor input plugin: http://docs.fluentd.org/articles/monitoring

Unfortunately, the documentation is pretty scant, and some of the useful, interesting endpoints and options are not documented. I've captured some of that missing information below, and shown how it can be used to monitor the Elasticsearch output plugin.

Endpoints

/api/plugins

Provides information about each plugin in a text based columnar format:
$ curl -s http://localhost:24220/api/plugins
plugin_id:object:1dce4b0        plugin_category:input   type:monitor_agent
output_plugin:false     retry_count:
plugin_id:object:11b4120        plugin_category:input   type:systemd    output_p
lugin:false     retry_count:
plugin_id:object:19fb914        plugin_category:output  type:rewrite_tag_filter
output_plugin:true      retry_count:
...

/api/plugins.json

Same as /api/plugins except in JSON format:
$ curl -s http://localhost:24220/api/plugins.json | python -mjson.tool
{
    "plugins": [
        {
            "config": {
                "@type": "monitor_agent",
                "bind": "0.0.0.0",
                "port": "24220"
            },
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"
        },
...

/api/config

Provides basic fluentd configuration information in text format:
$ curl -s http://localhost:24220/api/config
pid:19  ppid:1  config_path:/etc/fluent/fluent.conf     pid_file:       plugin_dirs:["/etc/fluent/plugin"]      log_path:

/api/config.json

Provides basic fluentd configuration information in JSON format:
$ curl -s http://localhost:24220/api/config.json | python -mjson.tool
{
    "config_path": "/etc/fluent/fluent.conf",
    "log_path": null,
    "pid": 19,
    "pid_file": null,
    "plugin_dirs": [
        "/etc/fluent/plugin"
    ],
    "ppid": 1
}

Query String Options

debug

For plugins, this will print all of the instance variables:
$ http://localhost:24220/api/plugins.json\?debug=1 | python -mjson.tool
{
    "plugins": [
        {
            "config": {
                "@type": "monitor_agent",
                "bind": "0.0.0.0",
                "port": "24220"
            },
            "instance_variables": {
                "bind": "0.0.0.0",
                "emit_config": false,
                "emit_interval": 60,
...

@type

Search for plugin by @type:
$ http://localhost:24220/api/plugins.json\?@type=monitor_agent | python -mjson.tool
{
    "plugins": [
        {
            "config": {
                "@type": "monitor_agent",
                "bind": "0.0.0.0",
                "port": "24220"
            },
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"
        }
    ]
}

@id

Search for plugin by @id. For example, in the above output, there is "plugin_id": "object:1dce4b0". Once you have identified the id, you can use that to display only the information for that particular id:
$ http://localhost:24220/api/plugins.json\?@id=object:1dce4b0 | python -mjson.tool
{
    "plugins": [
        {
            "config": {
                "@type": "monitor_agent",
                "bind": "0.0.0.0",
                "port": "24220"
            },
            "output_plugin": false,
            "plugin_category": "input",
            "plugin_id": "object:1dce4b0",
            "retry_count": null,
            "type": "monitor_agent"
        }
    ]
}

tag

Match the tag and get the info from the matched output plugin. Only works on output plugins. I unfortunately don't have an example, but I suppose you could use something like this to find the output plugins which have a match block which has a match for **_sendtoforwarder_**:
$ http://localhost:24220/api/plugins.json\?tag=prefix_sendtoforwarder_suffix | python -mjson.tool
{
    "plugins": [
        {
...

Debugging the Fluentd Elasticsearch plugin


First, identify the output plugin in question to get the plugin id:
$ http://localhost:24220/api/plugins.json\?@type=elasticsearch_dynamic | python -mjson.tool
{
    "plugins": [
        {
            "buffer_queue_length": 0,
            "buffer_total_queued_size": 0,
            "config": {
                "@type": "elasticsearch_dynamic",
...
                "index_name": ".operations.${record['@timestamp'].nil? ? Time.at
(time).getutc.strftime(@logstash_dateformat) : Time.parse(record['@timestamp']).
getutc.strftime(@logstash_dateformat)}",
...
            "plugin_id": "object:1b4cc64",
...

This is the one I'm looking for, which has a plugin id of object:1b4cc64. Next, I can use the @id parameter in conjunction with the debug one to get some interesting statistics:
$ http://localhost:24220/api/plugins.json\?@id=object:1b4cc64\&debug=1 | \
  python -mjson.tool | \
  egrep 'buffer_total_queued_size|emit_count'
            "buffer_total_queued_size": 0,
                "emit_count": 3164,

I can even put this in a simple loop to see how the queue size and emit count change over time:
$ while true ; do
  date
  http://localhost:24220/api/plugins.json\?@id=object:1b4cc64\&debug=1 | \
    python -mjson.tool | egrep 'buffer_total_queued_size|emit_count'
  sleep 1
done
Wed Dec  7 23:56:18 UTC 2016
            "buffer_total_queued_size": 0,
                "emit_count": 3318,
Wed Dec  7 23:56:21 UTC 2016
            "buffer_total_queued_size": 1654,
                "emit_count": 3322,
Wed Dec  7 23:56:23 UTC 2016
            "buffer_total_queued_size": 2146,
                "emit_count": 3324,
Wed Dec  7 23:56:25 UTC 2016
            "buffer_total_queued_size": 0,
                "emit_count": 3326,

This tells me that the plugin is working, the queues are being flushed regularly, and the emit count (roughly, the number of times fluentd flushes the queued outputs, the number of times a request is made to Elasticsearch) is steadily increasing.

Comments