Ubuntu
maas package

maas doesn't return zookeeper instances for newly provision environment

Bug #1233831 reported by Peter Petrakis
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
maas (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

The problem:
I can clearly see that a machine is allocated in response to the bootstrap request and that the
same machine is destroyed when the env is trashed, yet, I can't get a handle on the environment.

Odd thing is this problem cropped up all of a sudden. I've been provisioning via juju
maybe 5-6 times a day due to the hardware speed over the last week and it just stops working.
I've tried restart the processes, upgrading to latest maas, and rebooting. It just stays in this
state. I can't use it for anything.

ubuntu@bysen:~$ juju bootstrap --constraints arch=arm
2013-10-01 15:49:53,466 INFO Bootstrapping environment 'maas' (origin: distro type: maas)...
2013-10-01 15:49:56,714 INFO 'bootstrap' command finished successfully

ubuntu@bysen:~$ juju status
2013-10-01 15:50:37,633 INFO Connecting to environment...
2013-10-01 15:50:38,215 ERROR juju environment not found: is the environment bootstrapped?

ubuntu@bysen:~$ juju destroy-environment
WARNING: this command will destroy the 'maas' environment (type: maas).
This includes all machines, services, data, and other resources. Continue [y/N] y
2013-10-01 15:50:46,997 INFO Destroying environment 'maas' (type: maas)...
2013-10-01 15:50:50,208 INFO 'destroy_environment' command finished successfully

tcpdump (see attached) shows that nothing is being returned in response to the
status call.

13:51:06.061698 IP bysen.local.53598 > bysen.local.http: Flags [P.], seq 1:394, ack 1, win 342, options [nop,nop,TS val 18200601 ecr 18200601], length 393
E...D.@.@...
. .
. ..^.P@....~.s...VYu.....
........GET /MAAS/api/1.0/files/?filename=provider-state&op=get HTTP/1.0
Host: 10.229.32.254
User-Agent: Twisted PageGetter
Authorization: OAuth realm="", oauth_nonce="39826450", oauth_timestamp="1380649866", oauth_consumer_key="DpGkmSKnKzGyJRW8QB", oauth_signature_method="PLAINTEXT", oauth_version="1.0", oauth_token="UUNw9cT2xh3rU8Fbq9", oauth_signature="%26bMECDATsYLtATZzG5n8dsvr8X3jUyX49"

13:51:06.061745 IP bysen.local.http > bysen.local.53598: Flags [.], ack 394, win 350, options [nop,nop,TS val 18200601 ecr 18200601], length 0
E..4oj@.@.s.
. .
. ..P.^.~.s@......^W......
........
13:51:06.648551 IP bysen.local.http > bysen.local.53598: Flags [P.], seq 1:214, ack 394, win 350, options [nop,nop,TS val 18200748 ecr 18200601], length 213
E.. ok@.@.r.
. .
. ..P.^.~.s@......^X......
........HTTP/1.1 200 OK
Date: Tue, 01 Oct 2013 17:51:06 GMT
Server: Apache/2.2.22 (Ubuntu)
Vary: Authorization,Cookie,Accept-Encoding
Content-Length: 3
Connection: close
Content-Type: text/html; charset=utf-8

{}

To compare, I traced my other maas, it should be returning something like this.

Date: Tue, 01 Oct 2013 19:27:16 GMT
Server: Apache/2.2.22 (Ubuntu)
Vary: Authorization,Cookie,Accept-Encoding
Connection: close
Content-Type: text/html; charset=utf-8

zookeeper-instances: [/MAAS/api/1.0/nodes/node-845efb14-bf1e-11e2-a3df-525400052fb9/]

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: maas 1.3+bzr1461+dfsg-0ubuntu2.2
ProcVersionSignature: Ubuntu 3.8.0-30.44-generic 3.8.13.6
Uname: Linux 3.8.0-30-generic x86_64
ApportVersion: 2.9.2-0ubuntu8.3
Architecture: amd64
Date: Tue Oct 1 15:35:56 2013
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: maas
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :
Revision history for this message
Gavin Panella (allenap) wrote :

This looks like Juju is not writing the provider-state file correctly. You're using PyJuju. Have you tried using juju-core instead? That might help us isolate the problem to either Juju or MAAS.

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :
Download full text (3.8 KiB)

juju-core appears to be a non-starter.

ubuntu@bysen:~$ juju -vvv bootstrap --constraints arch=arm -e maas
2013-10-01 21:12:17 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-01 21:12:17 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-01 21:12:17 INFO juju.environs.tools tools.go:37 filtering tools by series: quantal
2013-10-01 21:12:17 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-01 21:12:34 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"quantal", Arch:"arm"}
2013-10-01 21:12:34 INFO juju.environs.sync sync.go:63 listing available tools
listing available tools
2013-10-01 21:12:35 INFO juju.environs.sync sync.go:76 found 32 tools
found 32 tools
2013-10-01 21:12:35 INFO juju.environs.sync sync.go:80 found 9 recent tools (version 1.14.1)
found 9 recent tools (version 1.14.1)
2013-10-01 21:12:35 INFO juju.environs.sync sync.go:86 listing target bucket
listing target bucket
2013-10-01 21:12:52 INFO juju.environs.sync sync.go:112 found 8 tools in target; 1 tools to be copied
found 8 tools in target; 1 tools to be copied
2013-10-01 21:12:52 INFO juju.environs.sync sync.go:132 copying 1.14.1-saucy-i386 from https://juju-dist.s3.amazonaws.com/tools/juju-1.14.1-saucy-i386.tgz
copying 1.14.1-saucy-i386 from https://juju-dist.s3.amazonaws.com/tools/juju-1.14.1-saucy-i386.tgz
2013-10-01 21:12:52 INFO juju.environs.sync sync.go:146 copying tools/juju-1.14.1-saucy-i386.tgz
copying tools/juju-1.14.1-saucy-i386.tgz
2013-10-01 21:13:04 INFO juju.environs.sync sync.go:159 downloaded tools/juju-1.14.1-saucy-i386.tgz (4081kB), uploading
downloaded tools/juju-1.14.1-saucy-i386.tgz (4081kB), uploading
2013-10-01 21:13:04 INFO juju.environs.sync sync.go:160 download 4081kB, uploading
download 4081kB, uploading
2013-10-01 21:13:05 INFO juju.environs.sync sync.go:117 copied 1 tools
copied 1 tools
2013-10-01 21:13:05 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-01 21:13:05 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-01 21:13:05 INFO juju.environs.tools tools.go:37 filtering tools by series: quantal
2013-10-01 21:13:05 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-01 21:13:24 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"quantal", Arch:"arm"}
2013-10-01 21:13:24 ERROR juju supercommand.go:282 command failed: no matching tools available
error: no matching tools available

It's also doing something weird, complaining about a environment that isn't defined and doesn't exist.
Completely ignored "default:" in my .config too.

ubuntu@bysen:~$ juju -vvv bootstrap --constraints arch=arm
2013-10-01 21:14:30 ERROR juju supercommand.go:282 command failed: unknown environment "cabs-pts-cx-maas"
error: unknown environment "cabs-pts-cx-maas"

ubuntu@bysen:~$ cat ~/.juju/environments.yaml
default: maas
environments:
 maas: # admin account
  type: maas
  maas-server: http://10.229.32.254:80/MAAS
  ma...

Read more...

Revision history for this message
John A Meinel (jameinel) wrote :

We only have builds of juju-core on Arm for Saucy (1.14.0 and 1.14.1 both have official juju tools builds for saucy-1.14.1-armhf.tgz)

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

Do I literally need to be on saucy to get these tools? I pinned the saucy version and it still doesn't work.

ubuntu@bysen:~$ dpkg -l juju-core
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============================-====================-====================-==================================================================
ii juju-core 1.14.1-0ubuntu1 amd64 Juju is devops distilled - client

ubuntu@bysen:~$ juju version
1.14.1-raring-amd64

ubuntu@bysen:~$ juju -vvv bootstrap --constraints arch=arm -e maas
2013-10-02 11:04:56 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 11:04:56 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 11:04:56 INFO juju.environs.tools tools.go:37 filtering tools by series: quantal
2013-10-02 11:04:56 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 11:05:13 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"quantal", Arch:"arm"}
2013-10-02 11:05:13 INFO juju.environs.sync sync.go:63 listing available tools
listing available tools
2013-10-02 11:05:15 INFO juju.environs.sync sync.go:76 found 32 tools
found 32 tools
2013-10-02 11:05:15 INFO juju.environs.sync sync.go:80 found 9 recent tools (version 1.14.1)
found 9 recent tools (version 1.14.1)
2013-10-02 11:05:15 INFO juju.environs.sync sync.go:86 listing target bucket
listing target bucket
2013-10-02 11:05:33 INFO juju.environs.sync sync.go:112 found 9 tools in target; 0 tools to be copied
found 9 tools in target; 0 tools to be copied
2013-10-02 11:05:33 INFO juju.environs.sync sync.go:117 copied 0 tools
copied 0 tools
2013-10-02 11:05:33 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 11:05:33 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 11:05:33 INFO juju.environs.tools tools.go:37 filtering tools by series: quantal
2013-10-02 11:05:33 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 11:05:51 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"quantal", Arch:"arm"}
2013-10-02 11:05:51 ERROR juju supercommand.go:282 command failed: no matching tools available
error: no matching tools available

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

Changed the env series, same result.

ubuntu@bysen:~$ juju -vvv bootstrap --constraints arch=arm -e maas
2013-10-02 11:18:08 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 11:18:08 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 11:18:08 INFO juju.environs.tools tools.go:37 filtering tools by series: saucy
2013-10-02 11:18:08 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 11:18:26 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"saucy", Arch:"arm"}
2013-10-02 11:18:26 INFO juju.environs.sync sync.go:63 listing available tools
listing available tools
2013-10-02 11:18:27 INFO juju.environs.sync sync.go:76 found 32 tools
found 32 tools
2013-10-02 11:18:27 INFO juju.environs.sync sync.go:80 found 9 recent tools (version 1.14.1)
found 9 recent tools (version 1.14.1)
2013-10-02 11:18:27 INFO juju.environs.sync sync.go:86 listing target bucket
listing target bucket
2013-10-02 11:18:45 INFO juju.environs.sync sync.go:112 found 9 tools in target; 0 tools to be copied
found 9 tools in target; 0 tools to be copied
2013-10-02 11:18:45 INFO juju.environs.sync sync.go:117 copied 0 tools
copied 0 tools
2013-10-02 11:18:45 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 11:18:45 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 11:18:45 INFO juju.environs.tools tools.go:37 filtering tools by series: saucy
2013-10-02 11:18:45 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 11:19:03 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"saucy", Arch:"arm"}
2013-10-02 11:19:03 ERROR juju supercommand.go:282 command failed: no matching tools available
error: no matching tools available

Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

@John, I don't dispute that the tools aren't there, but I'm starting to think juju-core
 isn't able to find it.

I created a saucy VM to try and drive this MAAS, it still won't find the arm tools.

@John (anyone), have you actually seen this work?

ubuntu@saucy:~$ uname -a
Linux saucy 3.11.0-8-generic #15-Ubuntu SMP Fri Sep 20 04:11:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@saucy:~$ juju version
1.14.1-saucy-amd64

ubuntu@saucy:~$ lsb_release --all
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Saucy Salamander (development branch)
Release: 13.10
Codename: saucy

ubuntu@saucy:~$ juju -vvv bootstrap --constraints arch=arm
2013-10-02 17:53:02 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 17:53:02 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 17:53:02 INFO juju.environs.tools tools.go:37 filtering tools by series: saucy
2013-10-02 17:53:02 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 17:57:28 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"saucy", Arch:"arm"}
2013-10-02 17:57:28 INFO juju.environs.sync sync.go:63 listing available tools
listing available tools
2013-10-02 17:57:29 INFO juju.environs.sync sync.go:76 found 32 tools
found 32 tools
2013-10-02 17:57:29 INFO juju.environs.sync sync.go:80 found 9 recent tools (version 1.14.1)
found 9 recent tools (version 1.14.1)
2013-10-02 17:57:29 INFO juju.environs.sync sync.go:86 listing target bucket
listing target bucket
2013-10-02 18:01:54 INFO juju.environs.sync sync.go:112 found 9 tools in target; 0 tools to be copied
found 9 tools in target; 0 tools to be copied
2013-10-02 18:01:54 INFO juju.environs.sync sync.go:117 copied 0 tools
copied 0 tools
2013-10-02 18:01:54 INFO juju.environs.tools tools.go:82 filtering tools by released version
2013-10-02 18:01:54 INFO juju.environs.tools tools.go:29 reading tools with major version 1
2013-10-02 18:01:54 INFO juju.environs.tools tools.go:37 filtering tools by series: saucy
2013-10-02 18:01:54 INFO juju.environs.tools tools.go:40 filtering tools by architecture: arm
2013-10-02 18:06:20 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:true, Number:version.Number{Major:0, Minor:0, Patch:0, Build:0}, Series:"saucy", Arch:"arm"}
2013-10-02 18:06:20 ERROR juju supercommand.go:282 command failed: no matching tools available
error: no matching tools available

Andres Rodriguez (andreserl)
Changed in maas (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.