devstack

rsyslog needs to have ratelimiting disabled

Bug #1024487 reported by Jay Pipes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
Fix Released
Undecided
Kevin Carter

Bug Description

Seeing messages like this in syslog.txt artifacts:

rsyslogd-2177: imuxsock begins to drop messages from pid 27879 due to rate-limiting

And then later in the log file:

rsyslogd-2177: imuxsock lost 51 messages from pid 27879 due to rate-limiting

Not sure how to do it, but we need to disable ratelimiting in rsyslog for the devstack builders.

Revision history for this message
Jay Pipes (jaypipes) wrote :

Setting to Low since we have the screen log files individually we can check.

Changed in openstack-ci:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
funicorn (funicorn) wrote :

I see a lot of this in /var/log/syslog

Any progress on this issue ?

Revision history for this message
Clark Boylan (cboylan) wrote :

In theory a change similar to https://github.com/openstack-infra/config/commit/1b7801d6298fbcc8e365a5736e354bb5ede88f6c could be made to stop ratelimiting.

tags: added: low-hanging-fruit
Changed in openstack-ci:
milestone: none → havana
Kevin Carter (kevin-carter)
Changed in openstack-ci:
assignee: nobody → Kevin Carter (kevin-carter)
status: Confirmed → In Progress
Revision history for this message
Kevin Carter (kevin-carter) wrote :

From what I know of rsyslog, the rate limiting was introduced by default in rsyslog < 7 and then later refactored in rsyslog v7+.
The overall rate-limiting functions can be removed and or set to 0 with the following variables in the /etc/rsyslog.conf:

$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0

I proposed a small change in the stack.sh which will set the rsyslog rate-limits to 0.
https://review.openstack.org/#/c/30783/

The change will make rsyslog handle all of the messages that are thrown at it.

Jeremy Stanley (fungi)
Changed in devstack:
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in devstack:
assignee: nobody → Kevin Carter (kevin-carter)
Revision history for this message
James E. Blair (corvus) wrote :

This looks good; I'm removing openstack-ci since it looks like this can be fixed in devstack.

no longer affects: openstack-ci
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/30783
Committed: http://github.com/openstack-dev/devstack/commit/e4859f0bd3aee92ccc588614925ed697d1ba06d2
Submitter: Jenkins
Branch: master

commit e4859f0bd3aee92ccc588614925ed697d1ba06d2
Author: cloudnull <email address hidden>
Date: Tue May 28 14:10:58 2013 -0500

    Set variables in rsyslog.conf to disable rate-limiting.

    Per Bug https://bugs.launchpad.net/openstack-ci/+bug/1024487, this will
    modify the rate-limiting setting for rsyslog.conf. If rate-limiting is
    being used the addition will set it to 0 otherwise the addition will
    add the variables to the configuration file and set them to 0.

    Implements: Variables "$SystemLogRateLimitBurst 0", and
    "$SystemLogRateLimitInterval 0" in "/etc/rsyslog.conf". These changes
    are persuent to what has been outlined in the rsyslog docs as found
    here : http://www.rsyslog.com/tag/SystemLogRateLimitInterval/

    Updated commit implements changes in code and placement post code
    review. Recent change implements the "SystemLogRateLimitBurst" in first
    if statement, which was "SystemLogRateLimitInterval" and wrong.

    Fixes: Bug 1024487
    Change-Id: I2637889cbe9e5e87bbfc0f1bb5047abae34d953c

Changed in devstack:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.