Canonical Juju

SetAPIHostPorts runs afoul of "state changing too quickly"

Bug #1613855 reported by Nate Finch
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Michael Foord
Canonical Juju 2.0-beta16
juju-core
Fix Released
Critical
Tim Penhey
juju-core 1.25.11

Bug Description

user reported:

"I did a reboot of my juju state nodes and when they restarted I lost the ability to run `juju --debug status` and now get an error `health ping failed: connection is shut down` and in my `juju debug-log` I am getting ` cannot publish API server addresses: cannot set API addresses: state changing too quickly; try again soon`."

This is on 2.0 beta 12, which is what currently ships in Xenial.

Checking the code, it looks like we're hitting the same type of bug as https://bugs.launchpad.net/juju-core/+bug/1537585 where we're asserting on structs. The key line looks like it's here: https://github.com/juju/juju/blob/fccb9b69e7ba3920cfbae31f35fdc608898c253f/state/address.go#L124

op := txn.Op{
 C: controllersC,
 Id: apiHostPortsKey,
 Assert: bson.D{{
  "apihostports", fromNetworkHostsPorts(existing),
 }},
}

Tim Penhey (thumper)
Changed in juju-core:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 2.0-beta16
Anastasia (anastasia-macmood)
Changed in juju-core:
assignee: nobody → Richard Harding (rharding)
Michael Foord (mfoord)
Changed in juju-core:
assignee: Richard Harding (rharding) → Michael Foord (mfoord)
Revision history for this message
Michael Foord (mfoord) wrote :

We need to stop asserting on the struct and change the assert to the specific values we care about. An easy fix.

Richard Harding (rharding)
Changed in juju-core:
status: Triaged → In Progress
Revision history for this message
Michael Foord (mfoord) wrote :

Actually, the host ports being compared in the Assert is a slice of slices - so converting this to a collection of asserts on individual fields of each member is annoying. I'm looking at comparing txn-revno instead.

Revision history for this message
Michael Foord (mfoord) wrote :

Fix proposed: https://github.com/juju/juju/pull/6041

Richard Harding (rharding)
Changed in juju-core:
status: In Progress → Fix Committed
Canonical Juju QA Bot (juju-qa-bot)
affects: juju-core → juju
Changed in juju:
milestone: 2.0-beta16 → none
milestone: none → 2.0-beta16
Curtis Hovey (sinzui)
Changed in juju:
status: Fix Committed → Fix Released
Revision history for this message
José Pekkarinen (koalinux) wrote :

Hi,

I hit it on juju 1.25, is there any plan to backport the fix?

Thanks!

José.

Revision history for this message
Anastasia (anastasia-macmood) wrote :

@Jose,

The fix referred to in comment # 3 can be backported to 1.25 tip.
However, at this stage we are not planning next 1.25.x release.

Are you building juju from github or are you using released versions?

Changed in juju-core:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 1.25.11
status: Triaged → Incomplete
Jorge Niedbalski (niedbalski)
Changed in juju-core:
status: Incomplete → Confirmed
Anastasia (anastasia-macmood)
Changed in juju-core:
status: Confirmed → Triaged
Tim Penhey (thumper)
Changed in juju-core:
assignee: nobody → Tim Penhey (thumper)
status: Triaged → In Progress
Revision history for this message
Anastasia (anastasia-macmood) wrote :

PR against 1.25: https://github.com/juju/juju/pull/7146

Anastasia (anastasia-macmood)
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.