Launchpad itself

Root page breaks when blog.launchpad.net is down

Bug #939055 reported by Robert Collins
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Triaged
High
Unassigned

Bug Description

Front page blog feed updates are done in-line with requests.

This fails hard when blog.launchpad.net is down.

It would be better to keep serving old content than fail; we need to either set a low timeout, or obtain the content asynchronously, and keep it cached in memcache indefinitely.

(Nagios alerted on all of LP when blog.l.n was down today)

See original description

Related branches

lp:~gz/launchpad/root_blog_feature_flag_939055
Benji York (community): Approve (code)
Diff: 194 lines (+90/-8)
5 files modified
lib/lp/app/browser/root.py (+9/-0)
lib/lp/app/browser/tests/test_launchpadroot.py (+64/-0)
lib/lp/app/stories/launchpad-root/front-pages.txt (+4/-2)
lib/lp/app/templates/root-index.pt (+7/-6)
lib/lp/services/features/flags.py (+6/-0)
Revision history for this message
Robert Collins (lifeless) wrote :

I'm going to escalate this, we need a fix in place for mid august. We need one of:
 - inject it into memcache asynchronously,
 - handle blog.l.n being gone very gracefully
 - allow it to be disabled via a featureflag

I suspect a featureflag is easier for now, which won't fully fix this bug but will give us the ability to do downtime on blog.l.n.

tags: added: escalated
Changed in launchpad:
assignee: nobody → John A Meinel (jameinel)
Revision history for this message
Robert Collins (lifeless) wrote :

@John, I've assigned you - this needs to jump the queue - see my internal mail for details. Feel free to delegate it, of course :)

John A Meinel (jameinel)
Changed in launchpad:
assignee: John A Meinel (jameinel) → Martin Packman (gz)
Martin Packman (gz)
Changed in launchpad:
status: Triaged → In Progress
Revision history for this message
Martin Packman (gz) wrote :

The new feature flag has been enabled in advance on production/staging/qastaging so as the change lands the current behaviour should be preserved.

Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r15805 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15805>.

tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Martin Packman (gz)
tags: added: qa-ok
removed: qa-needstesting
description: updated
summary: - front page blog feed updates are done in-line with requests
+ Root page breaks when blog.launchpad.net is down
Steve Kowalik (stevenk)
Changed in launchpad:
status: Fix Committed → Fix Released
Revision history for this message
Robert Collins (lifeless) wrote :

Current solution provides manual workaround, doesn't prevent the error happening if the flag isn't set.

Changed in launchpad:
status: Fix Released → Triaged
assignee: Martin Packman (gz) → nobody
Francis J. Lacoste (flacoste)
tags: removed: escalated
Revision history for this message
Martin Packman (gz) wrote :

From testing while working on this, the blog being down doesn't break launchpad anyway. Provided the connection is refused or an error is returned promptly, the page will work.

Given this flag apparently didn't need to get set over the weekend for the move and nothing seems to have exploded, is any kind of workaround here actually required?

Revision history for this message
William Grant (wgrant) wrote :

It did explode -- the front page timed out for the duration of the GR move. An error wasn't returned promptly, as the entire network was missing.

Revision history for this message
Stuart Bishop (stub) wrote : Re: [Bug 939055] Re: Root page breaks when blog.launchpad.net is down

On Mon, Aug 20, 2012 at 4:43 PM, Martin Packman
<email address hidden> wrote:
> >From testing while working on this, the blog being down doesn't break
> launchpad anyway. Provided the connection is refused or an error is
> returned promptly, the page will work.
>
> Given this flag apparently didn't need to get set over the weekend for
> the move and nothing seems to have exploded, is any kind of workaround
> here actually required?

The launchpad front page was down for a chunk of the weekend, so it did explode.

Given the Launchpad timeout constraints, I don't think we can
reasonably expect the blog to return in the subsecond time required.
Perhaps a background task to refresh the feed regularly, and Launchpad
loads it from local disk?

--
Stuart Bishop <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.