Launchpad itself

Timeout OOPS on multiple operations due to database lock contention

Bug #289037 reported by Sarah Kowalik
28
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Данило Шеган
Launchpad itself 2.1.11

Bug Description

This happened last sunday, multiple times during the week, and has happened again tonight. Analysis yesterday seemed to indicate that this was due to database locking issues triggered by the language pack export process.

Problems have been observed in at least:

* Soyuz / PPA
  - Builds are not being started
  - Current builds are listed as having started many hours ago, but aren't going anywhere
  - Nothing is showing as being uploaded
  - Queue accepting keeps timing out (sometimes also happens on cocoplum). (OOPS-1029F861)

* Bugs
  - Nominate for release keeps timing out. (OOPS-1029EC39, etc)

See original description
Matt Zimmerman (mdz)
description: updated
Revision history for this message
Matt Zimmerman (mdz) wrote : Re: Timeout OOPS on multiple operations (during langpack export?)

Confirmed, I've reproduced this trying to nominate bug 287941 for Intrepid: OOPS-1029EA40

Revision history for this message
Matt Zimmerman (mdz) wrote :

<stub> mdz: Should be ok now. My fault this time, or the lack of coffee when I did it this morning - the langpack export against staging picked up the wrong config and was actually running against the production db.

Changed in launchpad:
assignee: nobody → stub
importance: Undecided → Critical
status: New → Fix Released
Christian Reis (kiko)
Changed in launchpad:
assignee: stub → danilo
importance: Critical → Undecided
status: Fix Released → In Progress
Revision history for this message
Matt Zimmerman (mdz) wrote :

The underlying locking issue which caused this problem is now filed as bug 289050

Revision history for this message
Stuart Bishop (stub) wrote :

Regular language pack exports are currently switched off until updated code is rolled out to forster.

Changed in rosetta:
status: In Progress → Fix Committed
Revision history for this message
Данило Шеган (danilo) wrote :

Fixed in RF 7219. (we should cherrypick it to forster as well)

Changed in rosetta:
importance: Undecided → Critical
milestone: none → 2.1.11
Revision history for this message
Matt Zimmerman (mdz) wrote : Re: [Bug 289037] Re: Timeout OOPS on multiple operations due to database lock contention

On Mon, Oct 27, 2008 at 10:39:31AM -0000, Данило Шеган wrote:
> Fixed in RF 7219. (we should cherrypick it to forster as well)
>
> ** Changed in: rosetta
> Importance: Undecided => Critical
> Target: None => 2.1.11

We do not anticipate a need to roll any more language packs for the 8.10
release, and so it is more important to minimize the impact on production.

In other words, I would prefer that we not cherrypick this until after 8.10.

--
 - mdz

Revision history for this message
Данило Шеган (danilo) wrote :

Cherrypicking is separate from actually re-instating the crontab entries to run language-pack-exporter script. I believe it would be smarter to cherrypick the fix just in the case that even if crontab entries are put back by mistake (or miscommunication), it doesn't cause problems for everybody.

Revision history for this message
Данило Шеган (danilo) wrote :

language-pack-exporter is disabled, and we are not going to run it until the next roll-out (late November). Lowering priority as a result.

Changed in rosetta:
importance: Critical → High
Revision history for this message
Данило Шеган (danilo) wrote :

This has been cherrypicked, and language packs have been running without locking any tables (or at least for longer than a second or two).

Changed in rosetta:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.