all watcher api regression

Bug #1284690 reported by Peter Petrakis
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
juju-deployer
Fix Released
Critical
Unassigned
python-jujuclient
Fix Released
Critical
Unassigned
python-jujuclient (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

It's back. It's also inconsistent.

Initially spotted by 'ev', https://bugs.launchpad.net/juju-deployer/+bug/1269519/comments/12

ii juju-deployer 0.3.4-0ubuntu1 Deploy complex stacks of services using Juju
ii python-jujuclie 0.15-0ubuntu1 Python API client for juju
ii juju-core 1.17.3-0ubuntu1 Juju is devops distilled - client

daq-emitter is just an alias for the ganglia sub.

2014-02-25 15:26:17 Config specifies num units for subordinate: daq-emitter
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.3.4', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 127, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 225, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 196, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 172, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 264, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 690, in run
    for change_set in self.watch:
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 216, in next
    return super(TimeoutWatcher, self).next()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 178, in next
    'Id': self.watcher_id})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 145, in _rpc
    raise EnvError(result)
jujuclient.EnvError: <Env Error - Details:
 { u'Error': u'state watcher was stopped', u'RequestId': 9, u'Response': { }}
 >

Tags: hs-arm64
Revision history for this message
Kapil Thangavelu (hazmat) wrote :

So apparently mongo conn to state server goes awol sometimes, and everything has todo a forced reconnect. this has been happening frequently in juju 1.17.3 (at least the the third report i've had in the last week). There needs to be a deployer workaround at least for watches to detect this and auto-reconnect.

Changed in juju-deployer:
status: New → Confirmed
importance: Undecided → Critical
milestone: none → 0.3.5
Revision history for this message
Peter Petrakis (peter-petrakis) wrote :
Revision history for this message
Kapil Thangavelu (hazmat) wrote :

i've uploaded jujuclient 0.17.3 to pypi which fixes at a lower level.

James Page (james-page)
Changed in python-jujuclient (Ubuntu):
importance: Undecided → High
Changed in python-jujuclient:
importance: Undecided → Critical
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-jujuclient - 0.17.3-0ubuntu1

---------------
python-jujuclient (0.17.3-0ubuntu1) trusty; urgency=medium

  * New upstream release:
    - Avoid issues with 'state watcher was stopped' messages when
      processes are forced to re-connect to Juju MongoDB (LP: #1284690).
 -- James Page <email address hidden> Tue, 04 Mar 2014 18:00:18 +0000

Changed in python-jujuclient (Ubuntu Trusty):
status: New → Fix Released
Revision history for this message
Kapil Thangavelu (hazmat) wrote :

last fix for reconnect had to be refactored, didn't work in the field, uploaded jujuclient-0.17.4 to pypi

Revision history for this message
Kapil Thangavelu (hazmat) wrote :

uploaded jujuclient 0.17.5 to pypi to handle connection errors while stopping a watch

James Page (james-page)
Changed in python-jujuclient (Ubuntu Trusty):
status: Fix Released → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-jujuclient - 0.17.5-0ubuntu1

---------------
python-jujuclient (0.17.5-0ubuntu1) trusty; urgency=medium

  * New upstream release:
    - Rework fix for issue with 'state watcher has stopped' messages
      (LP: #1284690).
 -- James Page <email address hidden> Tue, 11 Mar 2014 16:40:57 +0000

Changed in python-jujuclient (Ubuntu Trusty):
status: In Progress → Fix Released
Sean Feole (sfeole)
tags: added: hs-arm64
Changed in juju-deployer:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.