Fluentd doesn't process multiline log pattern

Fix proposed to branch: master
Review: https://review.openstack.org/494634

Fix proposed to branch: master
Review: https://review.openstack.org/494635

Reviewed: https://review.openstack.org/494634
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=83baad304d359e17a5be3ad1e92f7cd31dc129e0
Submitter: Jenkins
Branch: master

commit 83baad304d359e17a5be3ad1e92f7cd31dc129e0
Author: Bertrand Lallau <email address hidden>
Date: Thu Aug 17 18:20:04 2017 +0200

    Fluentd: add fluent-plugin-concat:1.0.0 gem

    Actual Fluentd config doesn't manage multiline log pattern like:

    2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [req-34ea3f23-f780-4059-b911-2788ea4af7b2 - - - - -] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
    nova-network is deprecated, as are any related configuration options.
    ). Its value may be silently ignored in the future.

    This log will generate 1 Fluentd records for each line (3 records),
    this is really a wrong behaviour, and made Kibana interface unusable.

    This patch add fluent-plugin-concat in order to concatenate multiline
    log separated in multiple events.

    Change-Id: I83e583605b49b7c421109d662de1291f38bd5c07
    Closes-Bug: #1711399

Reviewed: https://review.openstack.org/494635
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=38a6d30411be2b8bb116bfb765b054309a3a9c91
Submitter: Jenkins
Branch: master

commit 38a6d30411be2b8bb116bfb765b054309a3a9c91
Author: Bertrand Lallau <email address hidden>
Date: Thu Aug 17 18:05:20 2017 +0200

    Fluentd: manage multiline log pattern

    Actual Fluentd config doesn't manage multiline log pattern like:

    2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [-] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
    nova-network is deprecated, as are any related configuration options.
    ). Its value may be silently ignored in the future.

    This log will generate 1 Fluentd event for each line (in previous
    example 3 events).
    This is really a wrong behaviour and made Kibana interface not friendly.
    This patch uses a multiline parser.

    Change-Id: Iee337645d1dc42876ec79be08ee3715f11a53d87
    Closes-Bug: #1711399

This issue was fixed in the openstack/kolla 5.0.0.0rc1 release candidate.

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/500834

Reviewed: https://review.openstack.org/500834
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=9392a35704d8a227e488c687f6bce281475e1a07
Submitter: Jenkins
Branch: stable/pike

commit 9392a35704d8a227e488c687f6bce281475e1a07
Author: Bertrand Lallau <email address hidden>
Date: Thu Aug 17 18:05:20 2017 +0200

    Fluentd: manage multiline log pattern

    Actual Fluentd config doesn't manage multiline log pattern like:

    2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [-] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
    nova-network is deprecated, as are any related configuration options.
    ). Its value may be silently ignored in the future.

    This log will generate 1 Fluentd event for each line (in previous
    example 3 events).
    This is really a wrong behaviour and made Kibana interface not friendly.
    This patch uses a multiline parser.

    Change-Id: Iee337645d1dc42876ec79be08ee3715f11a53d87
    Closes-Bug: #1711399
    (cherry picked from commit 38a6d30411be2b8bb116bfb765b054309a3a9c91)

This issue was fixed in the openstack/kolla-ansible 5.0.0.0rc2 release candidate.

This issue was fixed in the openstack/kolla-ansible 6.0.0.0b2 development milestone.