Ubuntu
rsyslog package

Message and memory corruption in rsyslog

Bug #1059592 reported by Stuart
268
This bug affects 2 people
Affects Status Importance Assigned to Milestone
rsyslog (Debian)
Fix Released
Unknown
rsyslog (Ubuntu)
Fix Released
High
Chris J Arges
Precise
Fix Released
High
Louis Bouchard
Quantal
Fix Released
High
Louis Bouchard
Raring
Fix Released
High
Chris J Arges

Bug Description

When using the RFC5424 format, I've seen a number of corruptions in the messages (missing messages and unprintable characters). I've also had rsyslog crash out with "*** glibc detected *** rsyslogd: corrupted double-linked list: ... ***". The message corruptions match this upstream bug - http://bugzilla.adiscon.com/show_bug.cgi?id=296

Based on the bug fixes to rsyslog between 5.8.6 and 5.10.0, it appears that there have been various race conditions and memory corruption issues that have since been fixed (5.8.6 is now nearly a year old).

In addition, rsyslog has some errors in the RFC5424 format that would also be fixed by an update to 5.10.0 (like the nil value for PID and the hostname in "last message repeated X times")

System information:
Ubuntu 12.04 LTS
rsyslog 5.8.6-1ubuntu8

--

Quantal/Precise SRU Justification

[Impact]
 * If rsyslogd is configured using the RFC5424 format, messages can become corrupted and rsyslogd can potentially crash.

[Test Case]
 * Enable RFC5424 format logging by adding the following to /etc/rsyslog.conf
   $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
 * Eventually crashes/corruption can occur. The reporter in the upstream bug also had dynamic file templates that could also irritate this bug.

[Regression Potential]
 * This patch is already present in upstream rsyslogd, and the patch cleanly backports to precise/quantal.
 * This patch modifies the message locking to ensure proper locking using the alternative format. Thus, testing using the traditional (default) format and RFC5424 format are needed.

[Other Info]
 * Upstream bug: http://bugzilla.adiscon.com/show_bug.cgi?id=296
 * Upstream patch: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=791b16ce06d75944e338a6e5fa14c0394bde6f1d

See original description
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Triaged: since the reporter points to the upstream bug/fix
High: Corrupted log messages are going to confuse lots of people and make debugging other things harder

As a precaution I marked as security; memory corruptions in rsyslog can't be good

security vulnerability: no → yes
Changed in rsyslog (Ubuntu):
importance: Undecided → High
status: New → Triaged
Chris J Arges (arges)
Changed in rsyslog (Ubuntu Precise):
importance: Undecided → High
Revision history for this message
Chris J Arges (arges) wrote :

Noticed this also affects quantal/raring. Getting patches ready.

Changed in rsyslog (Ubuntu Precise):
assignee: nobody → Chris J Arges (christopherarges)
Changed in rsyslog (Ubuntu Quantal):
importance: Undecided → High
assignee: nobody → Chris J Arges (christopherarges)
Changed in rsyslog (Ubuntu Raring):
assignee: nobody → Chris J Arges (christopherarges)
Changed in rsyslog (Ubuntu Precise):
status: New → In Progress
Changed in rsyslog (Ubuntu Quantal):
status: New → In Progress
Changed in rsyslog (Ubuntu Raring):
status: Triaged → In Progress
Revision history for this message
Chris J Arges (arges) wrote :

Branches linked for Precise/Quantal SRU. Raring could be fixed by a 5.8.7 > sync.

description: updated
Changed in rsyslog (Ubuntu Raring):
assignee: Chris J Arges (christopherarges) → nobody
Revision history for this message
Chris J Arges (arges) wrote :

submittodebian patch sent:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693910

Chris J Arges (arges)
description: updated
Bug Watch Updater (bug-watch-updater)
Changed in rsyslog (Debian):
status: Unknown → Fix Released
Revision history for this message
Sebastien Bacher (seb128) wrote :

@Chris: thanks for preparing the updates, do you think you could get the fix applied for raring as well? We can't SRU a fix before having it available in the current serie

Chris J Arges (arges)
Changed in rsyslog (Ubuntu Raring):
assignee: nobody → Chris J Arges (christopherarges)
Revision history for this message
Chris J Arges (arges) wrote :

@seb128
Linked a branch for raring. Thanks

Sebastien Bacher (seb128)
Changed in rsyslog (Ubuntu Raring):
status: In Progress → Fix Committed
Revision history for this message
Sebastien Bacher (seb128) wrote :

Thanks, I've uploaded to the 3 series

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu11

---------------
rsyslog (5.8.6-1ubuntu11) raring; urgency=low

  * debian/patches/101-fix-rfc5424-instabilities.patch:
    - bugfix: instabilities when using RFC5424 header fields (LP: #1059592)
 -- Chris J Arges <email address hidden> Tue, 04 Dec 2012 08:59:07 -0600

Changed in rsyslog (Ubuntu Raring):
status: Fix Committed → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Stuart, or anyone else affected,

Accepted rsyslog into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/5.8.6-1ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in rsyslog (Ubuntu Quantal):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in rsyslog (Ubuntu Precise):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

Hello Stuart, or anyone else affected,

Accepted rsyslog into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/5.8.6-1ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Brian Murray (brian-murray) wrote : [rsyslog/precise] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Brian Murray (brian-murray) wrote : [rsyslog/precise] possible regression found

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of rsyslog from precise-proposed was performed and bug 1092936 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1092936 "bot-stop-nagging". Thanks!

tags: added: verification-failed
Revision history for this message
Steve Langasek (vorlon) wrote :

reviewing bug #1092936 and the diff for this change, they don't appear to be related; the SRU version just happens to be the verison that bug #1092936 was reported against, there's not a causal relationship here.

tags: removed: verification-failed
Revision history for this message
Sebastien Bacher (seb128) wrote :

@Chris: is there any chance you guys test the SRU fix so the update doesn't go wasted?

Revision history for this message
Louis Bouchard (louis) wrote :

@seb128

I'll do my best to test that one tomorrow morning so we don't waste this one and hopefully I'll be able to reproduce the message

Revision history for this message
Chris J Arges (arges) wrote :

@seb128.
Yes, getting help from caribou on verifying this.

Revision history for this message
Louis Bouchard (louis) wrote :

After a few hours of stress tests on the original version, I'm not able to get the cited corruption.

I have installed the version in -proposed and will re-run the stress test overnight to confirm that no regression exist with the new version. If nothing is outlined, I will mark it as "verificaiton-done" tomorrow

Revision history for this message
Louis Bouchard (louis) wrote :

The new rsyslog package has been running overnight under stress test and is still performing as expected. I'm marking this one "verification-done" since no regression seems present.

tags: added: verification-done
removed: removal-candidate verification-needed
Changed in rsyslog (Ubuntu Precise):
assignee: Chris J Arges (arges) → Louis Bouchard (louis-bouchard)
Changed in rsyslog (Ubuntu Quantal):
assignee: Chris J Arges (arges) → Louis Bouchard (louis-bouchard)
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu9.1

---------------
rsyslog (5.8.6-1ubuntu9.1) quantal-proposed; urgency=low

  * debian/patches/101-fix-rfc5424-instabilities.patch:
    - bugfix: instabilities when using RFC5424 header fields (LP: #1059592)
 -- Chris J Arges <email address hidden> Wed, 21 Nov 2012 10:53:54 -0600

Changed in rsyslog (Ubuntu Quantal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu8.1

---------------
rsyslog (5.8.6-1ubuntu8.1) precise-proposed; urgency=low

  * debian/patches/101-fix-rfc5424-instabilities.patch:
    - bugfix: instabilities when using RFC5424 header fields (LP: #1059592)
 -- Chris J Arges <email address hidden> Wed, 21 Nov 2012 09:41:31 -0600

Changed in rsyslog (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.