Consistent retry failures on web UI

IntegrityError: duplicate key value violates unique constraint "buildpackagejob__build__key"

The build failed without removing the BPJ, BQ and Job? I am scared.

Indeed does sound scary... I'd initially hoped another possibility could have been that the records were removed, but the first retry OOPS may have left the build in an inconsistent state, but this doesn't seem possible either (build.retry() updates the buildstate before creating the new queue records).

I just got a chance to check the logs, and there's nothing of interest there either, other than the repetitive:

2010-04-09 11:42:25+0100 [QueryWithTimeoutProtocol,client] <floe:http://floe.buildd:8221/> marked as done. [4]

As to cause, at the moment all I can see is a number of things that could fail between setting the buildstate and deleting the related queue record etc.:

{{{
        self.buildstate = BuildStatus.FAILEDTOBUILD
        self.storeBuildInfo(librarian, slave_status)
        self.buildqueue_record.builder.cleanSlave()
        self.notify()
        self.buildqueue_record.destroySelf()
}}}

As to a temporary fix, I'll organise to have the offending records deleted first thing tomorrow (unless someone beats me to it).

On Monday 12 April 2010 17:16:30 Michael Nelson wrote:
> {{{
> self.buildstate = BuildStatus.FAILEDTOBUILD
> self.storeBuildInfo(librarian, slave_status)
> self.buildqueue_record.builder.cleanSlave()
> self.notify()
> self.buildqueue_record.destroySelf()
> }}}

One explanation if indeed this piece of code is getting called is that
something is doing a commit() in storeBuildInfo(), cleanSlave() or notify(),
before failing and aborting the rest of the txn.

Just for the record, the query to gather all the info:
https://pastebin.canonical.com/30473/

The result:
https://pastebin.canonical.com/30474/

And then deleting the offending records:
https://pastebin.canonical.com/30475/

And the build is now pending.

Did you get any more of these Scott?

I've removed the cron job (queue-builder) that kicked off a script that adds missing build records, it was conflicting with the build dispatcher in unpredictable ways.

If there were no more problems then I'll close this bug.

It's been intermittent, so it's hard to know. I'd say go ahead and close it and I'll file a new bug if it comes up again.

Ok thanks Scott.