Transient error /w MAAS provider: Unknown operation: 'list_allocated'.

Bug #1021861 reported by Adam Gandelman
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Triaged
High
Unassigned
pyjuju
New
Undecided
Unassigned

Bug Description

I've seen this with my own eyes once in the last couple of days when running 'juju status' but it seems to be happening sporadically in our scripted deployments.

After deploying a bunch of services, waiting for them to hit 'started' state, and adding a number of relations, I've noticed this in our logs:

2012-07-06 13:32:40,953 INFO Connecting to environment...
2012-07-06 13:32:41,781 INFO Connected to environment.
2012-07-06 13:32:42,055 INFO Added mysql-shared relation to all service units.
2012-07-06 13:32:42,055 INFO 'add_relation' command finished successfully
2012-07-06 13:32:47,432 INFO Connecting to environment...
2012-07-06 13:32:47,738 ERROR Cannot connect to environment: Unknown operation: 'list_allocated'.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 42, in run
    client = yield self._internal_connect(share)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 58, in _internal_connect
    candidates = yield self._provider.get_zookeeper_machines()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/providers/common/findzookeepers.py", line 32, in find_zookeepers
    machine = yield provider.get_machine(instance_id)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/providers/maas/provider.py", line 91, in get_machines
    instances = yield self.maas_client.get_nodes(instance_ids)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 69, in _process_error
    raise ProviderError(error.response)
ProviderError: Unknown operation: 'list_allocated'.
2012-07-06 13:32:47,739 ERROR Unknown operation: 'list_allocated'.
2012-07-06 13:32:53,109 INFO Connecting to environment...
2012-07-06 13:32:53,927 INFO Connected to environment.
2012-07-06 13:32:54,159 INFO Added rabbitmq relation to all service units.
2012-07-06 13:32:54,159 INFO 'add_relation' command finished successfully
2012-07-06 13:32:59,545 INFO Connecting to environment...

In this case, the mysql relation is the first of many to be added. The others seem to be added fine. I'm unsure where to look in MAAS's logfiles for any relevant traces on that end.

I've also been experiencing transient errors deploying services, though I'm not certain it is the same issue. I will update here if it turns out the same traceback is popping up around those failures.

Tags: provisioning
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

juju 0.5.1+bzr555-0juju1~precise1
maas 0.1+bzr482+dfsg-0ubuntu1
maas-provision-common 2.2.2-0ubuntu4

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Seems to be some fault in MAAS's use of Cobbler to list nodes? Also occasionally triggers during 'juju deploy', and maas.log reports:

ERROR 2012-07-19 01:28:46,933 maas.maasserver Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/decorators/vary.py", line 22, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 166, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 164, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 251, in dispatcher
    self, request, request.method, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 193, in perform_api_operation
    return method(handler, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 398, in release
    node.save()
  File "/usr/lib/python2.7/dist-packages/maasserver/models.py", line 114, in save
    return super(CommonInfo, self).save(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 570, in save_base
    created=(not record_exists), raw=raw, using=using)
  File "/usr/lib/python2.7/dist-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/lib/python2.7/dist-packages/maasserver/provisioning.py", line 499, in provision_post_save_Node
    papi.modify_nodes({instance.system_id: delta})
  File "/usr/lib/python2.7/dist-packages/maasserver/provisioning.py", line 254, in __call__
    raise friendly_fault
ExternalComponentException: Unknown problem encountered with the Cobbler server

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 1021861] Re: Transient error /w MAAS provider: Unknown operation: 'list_allocated'.

> ExternalComponentException: Unknown problem encountered with the Cobbler
> server

When it does this can you check the pserv.log please, it'll hopefully have
more detail.

Changed in maas:
status: New → Triaged
importance: Undecided → High
tags: added: provisioning
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Julian-

Looking through pserv.log, I see no entries relative to the timestamps in maas.log. Is there some way of increasing the verbosity of pserv.log logging?

Thanks

Revision history for this message
Julian Edwards (julian-edwards) wrote :

It should contain enough by default. That's very odd. OK, does the cobbler log have anything in it?

Revision history for this message
Julian Edwards (julian-edwards) wrote :

I've never seen this error, are you still seeing it? We're about to release a version without Cobbler so perhaps you can test with that when it comes out (or if you are feeling brave you can try trunk).

Revision history for this message
Diogo Matsubara (matsubara) wrote :

Laura recently triggered this bug using juju on quantal and maas - 0.1+bzr1063+dfsg-0+1099+98~ppa0~quantal1.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Diogo - that revision can't be right, it doesn't have cobbler in it.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Adam, was there anything in the cobbler log of note?

Revision history for this message
Francis J. Lacoste (flacoste) wrote :

That looks similar to bug 1058316 which we fixed (basically a problem with invalid oauth credentials). Marking as a duplicate, unbreak if that wasn't the case.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.