MAAS

Internal Server Error after installing MAAS from Trusty daily ISO

Bug #1298559 reported by Ryan Beisner
32
This bug affects 6 people
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Critical
Unassigned
maas (Ubuntu)
Fix Released
Critical
Andres Rodriguez
Trusty
Fix Released
Critical
Andres Rodriguez

Bug Description

After installing MAAS from ISO using the 'Create a new MAAS on this server' option, the MAAS frontend returns 'Internal Server Error'. Confirmed in a new virtual machine on an isolated network with no other DHCP service present. Possibly an issue with rabbitmq authentication or configuration (?).

Installation medium: Trusty Daily ISO 20140326

See also QA ISO Tracker 'Test-case name: maas/maas-002' on:
http://iso.qa.ubuntu.com/qatracker/milestones/308/builds/65451/testcases/1461/results

*** HTTP output:
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Apache/2.4.7 (Ubuntu) Server at 10.17.17.201 Port 80

*** Repeated entries from /var/log/maas/celery-region.log:
[2014-03-27 13:38:34,936: ERROR/Beat] beat: Connection error: [Errno 104] Connection reset by peer. Trying again in 32.0 seconds...
[2014-03-27 13:39:09,919: ERROR/MainProcess] consumer: Cannot connect to amqp://maas_workers@10.17.17.201:5672//maas_workers: [Errno 104] Connection reset by peer.
Trying again in 32.00 seconds...

*** /var/log/rabbitmq/rabbit@0-maas-node0.log:

=INFO REPORT==== 27-Mar-2014::13:42:30 ===
accepting AMQP connection <0.362.0> (127.0.0.1:44442 -> 127.0.0.1:5672)

=ERROR REPORT==== 27-Mar-2014::13:42:33 ===
closing AMQP connection <0.362.0> (127.0.0.1:44442 -> 127.0.0.1:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'maas_longpoll' - invalid credentials",
                             'connection.start_ok'}}

=INFO REPORT==== 27-Mar-2014::13:42:33 ===
accepting AMQP connection <0.365.0> (127.0.0.1:44443 -> 127.0.0.1:5672)

*** Repeated entries from /var/log/maas/txlongpoll.log:
2014-03-27 13:35:17-0500 [-] Starting factory <txlongpoll.client.AMQFactory instance at 0x7f79b09be290>
2014-03-27 13:35:17-0500 [-] Starting factory <txlongpoll.client.AMQFactory instance at 0x7f79b09be290>
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Unhandled error in Deferred:
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Unhandled Error
        Traceback (most recent call last):
        Failure: txamqp.client.Closed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]

2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] <twisted.internet.tcp.Connector instance at 0x7f79b09beb90> will retry in 0 seconds
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] <twisted.internet.tcp.Connector instance at 0x7f79b09beb90> will retry in 0 seconds
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Stopping factory <txlongpoll.client.AMQFactory instance at 0x7f79b09be290>
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Stopping factory <txlongpoll.client.AMQFactory instance at 0x7f79b09be290>
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Logged OOPS id OOPS-4258f0d407c032c62e32ab16d12d7ba0: No exception type: No exception value
2014-03-27 13:35:20-0500 [AMQClientWithCallback,client] Logged OOPS id OOPS-eb7e6f729bdc8b9931d9c9a41969c409: Closed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]

*** OOPS event:
rbeisner@0-maas-node0:/var/log/maas/oops/2014-03-27$ cat OOPS-eb7e6f729bdc8b9931d9c9a41969c409
0-maas-node0typeClosedid&OOPS-eb7e6f729bdc8b9931d9c9a41969c409value
[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]tb_text
Unhandled Error
Traceback (most recent call last):
Failure: txamqp.client.Closed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]

*** Repeated entries from /var/log/maas/pserv.log:

2014-03-27 13:39:15-0500 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://10.17.17.201/MAAS/rpc>
2014-03-27 13:39:15-0500 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://10.17.17.201/MAAS/rpc>
2014-03-27 13:39:15-0500 [HTTPPageGetter,client] Logged OOPS id OOPS-5f2ac431eb693077fb88cb0b92e21cb0: Error: 500 Internal Server Error
2014-03-27 13:40:19-0500 [-] Starting factory <HTTPClientFactory: http://10.17.17.201/MAAS/rpc>
2014-03-27 13:40:19-0500 [-] Starting factory <HTTPClientFactory: http://10.17.17.201/MAAS/rpc>
2014-03-27 13:40:35-0500 [HTTPPageGetter,client] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.web.error.Error: 500 Internal Server Error

*** Package & system info:
rbeisner@0-maas-node0:/usr/share/maas$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================================-=============================-============-===============================================================================
ii maas 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-cli 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Client Tool
ii maas-cluster-controller 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Cluster Controller
ii maas-common 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-dhcp 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - DHCP Configuration (meta-package)
ii maas-dns 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - DNS configuration (metapackage)
ii maas-region-controller 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-region-controller-min 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii python-django-maas 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - (django files)
ii python-maas-client 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS API Client - (python files)
ii python-maas-provisioningserver 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server

rbeisner@0-maas-node0:/etc$ uname -a
Linux 0-maas-node0 3.13.0-19-generic #40-Ubuntu SMP Mon Mar 24 02:36:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

rbeisner@0-maas-node0:/etc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Trusty Tahr (development branch)
Release: 14.04
Codename: trusty

*** from Apache error log:
[Thu Mar 27 13:59:49.533513 2014] [:error] [pid 1197:tid 140205665552128] [remote 10.17.17.201:15947] error: [Errno 104] Connection reset by peer
[Thu Mar 27 14:00:07.663804 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] mod_wsgi (pid=1198): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Thu Mar 27 14:00:07.663831 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] mod_wsgi (pid=1198): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Thu Mar 27 14:00:07.663862 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] Traceback (most recent call last):
[Thu Mar 27 14:00:07.663886 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/share/maas/wsgi.py", line 32, in <module>
[Thu Mar 27 14:00:07.663924 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] start_up()
[Thu Mar 27 14:00:07.663934 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 73, in start_up
[Thu Mar 27 14:00:07.663951 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] post_start_up()
[Thu Mar 27 14:00:07.663961 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 110, in post_start_up
[Thu Mar 27 14:00:07.663985 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] NodeGroup.objects.refresh_workers()
[Thu Mar 27 14:00:07.663995 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 117, in refresh_workers
[Thu Mar 27 14:00:07.664009 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] refresh_worker(nodegroup)
[Thu Mar 27 14:00:07.664017 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/maasserver/refresh_worker.py", line 41, in refresh_worker
[Thu Mar 27 14:00:07.664031 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] refresh_secrets.apply_async(queue=nodegroup.work_queue, kwargs=items)
[Thu Mar 27 14:00:07.664039 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/celery/app/task.py", line 554, in apply_async
[Thu Mar 27 14:00:07.664054 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] **dict(self._get_exec_options(), **options)
[Thu Mar 27 14:00:07.664062 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/celery/app/base.py", line 320, in send_task
[Thu Mar 27 14:00:07.664076 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] reply_to=reply_to or self.oid, **options
[Thu Mar 27 14:00:07.664084 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 299, in publish_task
[Thu Mar 27 14:00:07.664098 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] **kwargs
[Thu Mar 27 14:00:07.664106 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 166, in publish
[Thu Mar 27 14:00:07.664120 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] routing_key, mandatory, immediate, exchange, declare)
[Thu Mar 27 14:00:07.664128 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 461, in _ensured
[Thu Mar 27 14:00:07.664141 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] interval_max)
[Thu Mar 27 14:00:07.664149 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 373, in ensure_connection
[Thu Mar 27 14:00:07.664161 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] interval_start, interval_step, interval_max, callback)
[Thu Mar 27 14:00:07.664170 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 228, in retry_over_time
[Thu Mar 27 14:00:07.664183 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] return fun(*args, **kwargs)
[Thu Mar 27 14:00:07.664191 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 244, in connect
[Thu Mar 27 14:00:07.664203 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] return self.connection
[Thu Mar 27 14:00:07.664211 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 751, in connection
[Thu Mar 27 14:00:07.664223 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] self._connection = self._establish_connection()
[Thu Mar 27 14:00:07.664231 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 710, in _establish_connection
[Thu Mar 27 14:00:07.664248 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] conn = self.transport.establish_connection()
[Thu Mar 27 14:00:07.664256 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 111, in establish_connection
[Thu Mar 27 14:00:07.664269 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] conn = self.Connection(**opts)
[Thu Mar 27 14:00:07.664277 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 163, in __init__
[Thu Mar 27 14:00:07.664290 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] (10, 30), # tune
[Thu Mar 27 14:00:07.664298 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 73, in wait
[Thu Mar 27 14:00:07.664310 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] self.channel_id, allowed_methods)
[Thu Mar 27 14:00:07.664318 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 220, in _wait_method
[Thu Mar 27 14:00:07.664330 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] self.method_reader.read_method()
[Thu Mar 27 14:00:07.664339 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 195, in read_method
[Thu Mar 27 14:00:07.664351 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] raise m
[Thu Mar 27 14:00:07.664370 2014] [:error] [pid 1198:tid 140205665552128] [remote 10.17.17.201:15947] error: [Errno 104] Connection reset by peer

Related branches

lp:~andreserl/maas/fix_lp1298559
Andres Rodriguez (community): Approve
Diff: 41 lines (+7/-3)
3 files modified
debian/changelog (+5/-1)
debian/maas-region-controller-min.maas-region-celery.upstart (+1/-1)
debian/maas-region-controller-min.maas-txlongpoll.upstart (+1/-1)
lp:ubuntu/trusty-proposed/maas
Revision history for this message
Ryan Beisner (1chb1n) wrote :
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Take note: the MAAS frontend works as expected when installing from apt.

ie. do a basic Trusty install from ISO, then issue:
    sudo apt-get install maas maas-dhcp maas-dns

Following reboot, the MAAS http frontend is functional, and you can proceed to adding superuser, importing images, etc.

This seems to affect only the 'Create a new MAAS on this server' method.

Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1298559

tags: added: iso-testing
Revision history for this message
Diogo Matsubara (matsubara) wrote :

I just reproduced this behavior testing the latest Trusty ISO.

Changed in maas:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Ok this is not a bug in upstream MAAS but in the server ISO. Can you file the bug there please?

Changed in maas:
status: Triaged → Invalid
Andres Rodriguez (andreserl)
Changed in maas (Ubuntu):
assignee: nobody → Andres Rodriguez (andreserl)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in maas (Ubuntu):
status: New → Confirmed
Revision history for this message
Alberto Salvia Novella (es20490446e) wrote :

It has a severe impact on a large portion of Ubuntu users.

Changed in maas (Ubuntu):
importance: Undecided → Critical
status: Confirmed → Triaged
status: Triaged → In Progress
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Confirmed 'internal server error' issue exists with Trusty 2014-APR-09 daily ISO, when using the 'Create a new MAAS on this server' method.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 1.5+bzr2236-0ubuntu1

---------------
maas (1.5+bzr2236-0ubuntu1) trusty; urgency=medium

  * New upstream bugfix release.
    - Fix simplestreams path to use the new data source path. (LP: #1305118)
    - Fix commissioning_distro_series validation failure. (LP: #1304408)
    - Fix DB migration 0072_remove_ipmi_autodetec. (LP: #1302156)
    - Update documentation.
  * debian/maas-region-controller-min.maas-region-celery.upstart,
    debian/maas-region-controller-min.maas-txlongpoll.upstart: Fix ISO
    install by making sure rabbitmq user is really created if it does not
    exist (LP: #1298559)
 -- Andres Rodriguez <email address hidden> Wed, 09 Apr 2014 18:36:55 -0400

Changed in maas (Ubuntu Trusty):
status: In Progress → Fix Released
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Fix confirmed on Trusty server ISO 2014-APR-10. MAAS front end is functional after installation via 'Create a new MAAS on this server' method.

Thanks, Andres!

Revision history for this message
Dave Walker (davewalker) wrote :

Ryan, how is this possible... The upload that closed this bug isn't yet on an ISO?

Revision history for this message
James Page (james-page) wrote :

Dave - I expect the install pulled in updates which fixed this problem; Ryan - we'll need to verify again once the next set of images are created.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Dave - i'll re-test today with no internet. But the good news is, using 20140410 iso, you can now get a working MAAS front end.

James - will do on 20140411, again with no-internet.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Installing MAAS controller from ISO with no internet connection fails due to dependencies of the maas, maas-dns and maas-region-controller packages. pkg-sel bails out with 100.

If we do expect users to be able to deploy a MAAS controller on an island with no internet connection, then this is probably a separate bug. If we expect users to have internet access in order to resolve dependencies from repos during install, then I would not consider this a bug.

What is our use case expectation for MAAS regarding connectivity during install?

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Re-confirmed fix in a new install using Trusty iso 2014-apr-10. MAAS front end is functional on first boot.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Confirmed fix in a new install using Trusty iso 2014-apr-11 ... MAAS front end is functional on first boot.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

FYI - related bug regarding missing dependencies in a no-network scenario: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1172566

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Take note: the missing dependency claim is debunked. Dependencies are indeed met on the Trusty ISO. I confirmed this by successful installation of MAAS from ISO on a machine with a fully configured ip interface, but with no internet access. There is however, a separate MAAS installation problem when no default gateway is set. Updated bug 1172566.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.