kolla

Fluentd broken with OSS Elasticsearch

Bug #1954759 reported by Dr. Jens Harbott
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Dr. Jens Harbott

Bug Description

When fluentd is running with an OSS Elasticsearch version as backend, it fails with an error like:

2021-12-10 17:35:57 +0000 [error]: fluent/log.rb:372:error: unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not a supported distribution of Elasticsearch."
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-7.15.0/lib/elasticsearch.rb:85:in `verify_with_version_or_header'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-7.15.0/lib/elasticsearch.rb:66:in `verify_elasticsearch'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-7.15.0/lib/elasticsearch.rb:40:in `method_missing'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.15.0/lib/elasticsearch/api/actions/info.rb:37:in `info'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:490:in `block in handle_last_seen_es_major_version'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/elasticsearch_index_template.rb:52:in `retry_operate'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.4/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin.rb:187:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin/multi_output.rb:99:in `each'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin/multi_output.rb:99:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/plugin.rb:187:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:132:in `add_match'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:74:in `block in configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:64:in `each'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:64:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/root_agent.rb:149:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/engine.rb:105:in `configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/engine.rb:80:in `run_configure'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:716:in `block in run_worker'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:968:in `main_process'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:708:in `run_worker'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/bin/fluentd:15:in `<top (required)>'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/bin/fluentd:23:in `load'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /opt/td-agent/bin/fluentd:23:in `<top (required)>'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /usr/sbin/td-agent:15:in `load'
  2021-12-10 17:35:57 +0000 [error]: fluent/supervisor.rb:967:main_process: /usr/sbin/td-agent:15:in `<main>'

This is similar to https://bugs.launchpad.net/kolla/+bug/1941073, except that now the elasticsearch ruby gems are involved instead of the python modules.

The issue can be fixed by downgrading the gems that are delivered within the td-agent omnibus pkg:

(fluentd)[root@testbed-node-1 /]# td-agent-gem list|grep elas
elasticsearch (7.15.0)
elasticsearch-api (7.15.0)
elasticsearch-transport (7.15.0)
fluent-plugin-elasticsearch (5.1.4)
(fluentd)[root@testbed-node-1 /]# td-agent-gem install elasticsearch --version =7.13.0
Fetching elasticsearch-api-7.13.0.gem
Fetching elasticsearch-7.13.0.gem
Fetching elasticsearch-transport-7.13.0.gem
Successfully installed elasticsearch-transport-7.13.0
Successfully installed elasticsearch-api-7.13.0
Successfully installed elasticsearch-7.13.0
Parsing documentation for elasticsearch-transport-7.13.0
Installing ri documentation for elasticsearch-transport-7.13.0
Parsing documentation for elasticsearch-api-7.13.0
Installing ri documentation for elasticsearch-api-7.13.0
Parsing documentation for elasticsearch-7.13.0
Installing ri documentation for elasticsearch-7.13.0
Done installing documentation for elasticsearch-transport, elasticsearch-api, elasticsearch after 8 seconds
3 gems installed
(fluentd)[root@testbed-node-1 /]# td-agent-gem uninstall elasticsearch --version =7.15.0
Successfully uninstalled elasticsearch-7.15.0
(fluentd)[root@testbed-node-1 /]# td-agent-gem uninstall elasticsearch-api --version =7.15.0
Successfully uninstalled elasticsearch-api-7.15.0
(fluentd)[root@testbed-node-1 /]# td-agent-gem uninstall elasticsearch-transport --version =7.15.0
Successfully uninstalled elasticsearch-transport-7.15.0
(fluentd)[root@testbed-node-1 /]# td-agent-gem list|grep elas
elasticsearch (7.13.0)
elasticsearch-api (7.13.0)
elasticsearch-transport (7.13.0)
fluent-plugin-elasticsearch (5.1.4)
(fluentd)[root@testbed-node-1 /]#

cf. https://github.com/uken/fluent-plugin-elasticsearch/blob/master/README.Troubleshooting.md#cannot-send-events-to-elasticsearch

This is for Ubuntu source-based images in stable/wallaby, but likely affects other images, just the method for downgrading gems may vary.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/821695

Changed in kolla:
status: New → In Progress
Dr. Jens Harbott (j-harbott)
Changed in kolla:
assignee: nobody → Dr. Jens Harbott (j-harbott)
Mark Goddard (mgoddard)
Changed in kolla:
importance: Undecided → Medium
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

I'm inclined to close this as "invalid" as we (me and frickler) identified it's not the case with upstream images as we pin td-agent to 4.0.* which gives us elasticsearch 7.8.1. That said, frickler is working on td-agent unpinning which will require this to be fixed. We have a hypothesis that our previous issue with unpinned td-agent [1] was related to the upgraded elasticsearch gem.

[1] https://bugs.launchpad.net/kolla/+bug/1930867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/821695
Committed: https://opendev.org/openstack/kolla/commit/a8e6924beaea3e268435156f6bbc34812a69a01c
Submitter: "Zuul (22348)"
Branch: master

commit a8e6924beaea3e268435156f6bbc34812a69a01c
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823155
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/828560

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/828563

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/828560
Committed: https://opendev.org/openstack/kolla/commit/5c6eb173973a7781c8955115a6573ae59e45a479
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5c6eb173973a7781c8955115a6573ae59e45a479
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    Backport includes 711ec0829a95a12ae9e290aeb3837b457e1dd215 [1].

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    [1] I353ef9b6b8da0e726c17ad2e06995d3690731e1d
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Closes-Bug: 1960444
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Co-Authored-By: Radosław Piliszek <email address hidden>
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
    (cherry picked from commit a8e6924beaea3e268435156f6bbc34812a69a01c)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/828563
Committed: https://opendev.org/openstack/kolla/commit/3bbc5b3299c69ec56e57d63bdb487121742a2730
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3bbc5b3299c69ec56e57d63bdb487121742a2730
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    Backport includes 711ec0829a95a12ae9e290aeb3837b457e1dd215 [1].

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    [1] I353ef9b6b8da0e726c17ad2e06995d3690731e1d
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Closes-Bug: 1960444
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Co-Authored-By: Radosław Piliszek <email address hidden>
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
    (cherry picked from commit a8e6924beaea3e268435156f6bbc34812a69a01c)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 14.0.0.0rc1

This issue was fixed in the openstack/kolla 14.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 12.2.0

This issue was fixed in the openstack/kolla 12.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 13.1.0

This issue was fixed in the openstack/kolla 13.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.