RPC connection failed: info = yield self._fetch_rpc_info(info_url) twisted.web.error.Error: 500 INTERNAL SERVER ERROR 2017-01-19 12:15:59 -: [info] Received SIGTERM, shutting down. 2017-01-19 12:17:20 -: [info] twistd 16.0.0 (/usr/bin/python3 3.5.2) starting up. 2017-01-19 12:17:20 -: [info] reactor class: twisted.internet.epollreactor.EPollReactor. 2017-01-19 12:40:55 -: [info] Received SIGTERM, shutting down. 2017-01-19 12:42:27 -: [info] twistd 16.0.0 (/usr/bin/python3 3.5.2) starting up. 2017-01-19 12:42:27 -: [info] reactor class: twisted.internet.epollreactor.EPollReactor. 2017-01-19 12:42:28 provisioningserver.rpc.clusterservice: [info] Region not available: Connection was refused by other side: 111: Connection refused. (While requesting RPC info at b'http://[::ffff:192.168.199.2]:5240/MAAS/rpc/'). 2017-01-19 12:42:29 provisioningserver.rpc.clusterservice: [info] Region not available: Connection was refused by other side: 111: Connection refused. (While requesting RPC info at b'http://[::ffff:192.168.199.2]:5240/MAAS/rpc/'). Port is alive: root@juju-rack2:~# netstat -untap | grep 5240 tcp6 0 0 :::5240 :::* LISTEN 1235/python3 tcp6 0 0 :::5240 :::* LISTEN 1217/python3 tcp6 0 0 :::5240 :::* LISTEN 1203/python3 tcp6 0 0 :::5240 :::* LISTEN 1211/python3 tcp6 0 0 192.168.199.2:5240 192.168.199.2:39136 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39184 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39172 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39308 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39234 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39246 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39010 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39096 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39210 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39222 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:38986 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39148 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.2:39060 TIME_WAIT - tcp6 0 0 192.168.199.2:5240 192.168.199.100:53545 ESTABLISHED 1211/python3 tcp6 0 0 192.168.199.2:5240 192.168.199.2:39022 TIME_WAIT - ... MaaS RPC service appears to be down according to /var/log/maas/maas.log: Jan 19 11:54:32 juju-rack2 maas.bootsources: [info] Updated boot sources cache. Jan 19 11:54:32 juju-rack2 maas.bootresources: [info] Started importing of boot images from 1 source(s). Jan 19 11:54:32 juju-rack2 maas.import-images: [info] Downloading image descriptions from http://images.maas.io/ephemeral-v3/daily/ Jan 19 11:54:43 juju-rack2 maas.bootresources: [info] Importing images from source: http://images.maas.io/ephemeral-v3/daily/ Jan 19 11:55:26 juju-rack2 maas.service_monitor_service: [error] Can't update service statuses, no RPC connection to region. Checking the services: root@juju-rack2:/var/log/maas# service maas-rackd status ● maas-rackd.service - MAAS Rack Controller Loaded: loaded (/lib/systemd/system/maas-rackd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2017-01-19 12:41:56 EST; 33min ago Docs: https://maas.io/ Process: 1249 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd6.conf (code=exited, status=0/SUCCESS) Process: 1243 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.conf (code=exited, status=0/SUCCESS) Process: 1233 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.sock (code=exited, status=0/SUCCESS) Main PID: 1255 (sh) CGroup: /system.slice/maas-rackd.service ├─1255 /bin/sh -c exec /usr/bin/authbind --deep /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver ├─1257 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-rackd └─1258 tee -a /var/log/maas/rackd.log Jan 19 13:14:58 juju-rack2.home.spyderdyne.net maas.service_monitor_service[1257]: [error] Can't update service statuses, no RPC conn Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9120]: maas : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]: maas : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9120]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9122]: maas : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9122]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]: pam_unix(sudo:session): session closed for user root Jan 19 13:15:28 juju-rack2.home.spyderdyne.net sudo[9122]: pam_unix(sudo:session): session closed for user root Jan 19 13:15:28 juju-rack2.home.spyderdyne.net sudo[9120]: pam_unix(sudo:session): session closed for user root root@juju-rack2:/var/log/maas# ps aux | grep maas maas 1200 0.0 0.0 1548 340 ? Ss 12:42 0:00 /bin/sh -c exec twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a $LOGFILE maas 1202 0.0 0.0 1548 336 ? Ss 12:42 0:00 /bin/sh -c exec twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a $LOGFILE maas 1203 7.9 7.4 299712 70540 ? Sl 12:42 2:53 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond maas 1204 0.0 0.0 3508 184 ? S 12:42 0:00 tee -a /var/log/maas/regiond.log maas 1205 0.0 0.0 1548 328 ? Ss 12:42 0:00 /bin/sh -c exec twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a $LOGFILE maas 1211 13.9 7.9 205604 75444 ? Sl 12:42 5:03 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond maas 1212 0.0 0.0 3508 116 ? S 12:42 0:00 tee -a /var/log/maas/regiond.log maas 1217 6.3 6.7 135016 63520 ? Sl 12:42 2:19 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond maas 1218 0.0 0.0 3508 188 ? S 12:42 0:00 tee -a /var/log/maas/regiond.log maas 1229 0.0 0.0 1548 340 ? Ss 12:42 0:00 /bin/sh -c exec twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a $LOGFILE maas 1235 6.2 6.7 134932 63724 ? Sl 12:42 2:16 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-regiond maas 1236 0.0 0.0 3508 116 ? S 12:42 0:00 tee -a /var/log/maas/regiond.log maas 1255 0.0 0.0 1548 328 ? Ss 12:42 0:00 /bin/sh -c exec /usr/bin/authbind --deep /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-rackd 2>&1 | tee -a $LOGFILE maas 1257 3.3 5.1 97312 48692 ? Sl 12:42 1:14 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile= --logger=provisioningserver.logger.EventLogger maas-rackd maas 1258 0.0 0.0 3508 124 ? S 12:42 0:00 tee -a /var/log/maas/rackd.log root 1297 0.0 0.3 41144 2860 ? Ss 12:42 0:00 /usr/sbin/squid -Y -f /var/lib/maas/maas-proxy.conf proxy 1299 0.0 0.9 46508 9240 ? S 12:42 0:01 (squid-1) -Y -f /var/lib/maas/maas-proxy.conf postgres 1445 0.0 0.7 190732 7468 ? Ss 12:42 0:00 postgres: maas maasdb 127.0.0.1(55174) idle postgres 1451 0.0 0.7 190732 7540 ? Ss 12:42 0:00 postgres: maas maasdb 127.0.0.1(55180) idle postgres 1458 0.0 0.7 190732 7540 ? Ss 12:42 0:00 postgres: maas maasdb 127.0.0.1(55186) idle postgres 1560 0.0 0.7 190732 7560 ? Ss 12:42 0:00 postgres: maas maasdb 127.0.0.1(55236) idle root 9887 0.0 0.0 4244 564 pts/0 S+ 13:18 0:00 grep --color=auto maas postgres 9888 0.0 1.0 190912 10164 ? Ss 13:18 0:00 postgres: maas maasdb 127.0.0.1(40692) idle in transaction postgres 9889 0.0 1.0 190896 9708 ? Ss 13:18 0:00 postgres: maas maasdb 127.0.0.1(40694) idle in transaction I found a hostname mismatch in /var/log/maas/regiond.log: django.core.exceptions.ValidationError: {'hostname': ['Nonexistant domain.']} 2017-01-19 11:47:10 twisted.python.log: [info] ::ffff:192.168.199.2 - - [19/Jan/2017:16:47:10 +0000] "GET /MAAS/static/assets/images/logos/favicon.ico HTTP/1.1" 200 15663 "http://192.168.199.2:5240/MAAS/accounts/login/?next=%2FMAAS%2F" "Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36" 2017-01-19 11:47:10 maasserver.rpc.regionservice: [critical] Failed to update regiond's process and endpoints; juju-rack2.home.spyderdyne.net:pid=15414 record's may be out of date Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop self.runUntilCurrent() File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 798, in runUntilCurrent f(*a, **kw) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback self._startRunCallbacks(fail) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks self._runCallbacks() --- --- File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 1021, in ignoreCancellation failure.trap(defer.CancelledError) File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 342, in trap self.raiseException() File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 368, in raiseException raise self.value.with_traceback(self.tb) File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext result = inContext.theWork() File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in inContext.theWork = lambda: context.call(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 857, in callInContext return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 225, in wrapper result = func(*args, **kwargs) File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 162, in call_with_lock return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction return func_outside_txn(*args, **kwargs) File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier return func(*args, **kwargs) File "/usr/lib/python3.5/contextlib.py", line 30, in inner return func(*args, **kwds) File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line 1306, in update region_obj.save(update_fields=update_fields) File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 1485, in save super(Node, self).save(*args, **kwargs) File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 28, in save self.full_clean() File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1171, in full_clean raise ValidationError(errors) I attempted to rejoin the rack controller to the region: root@juju-rack2:/var/log/maas# dpkg-reconfigure maas-rack-controller root@juju-rack2:/var/log/maas# maas-rack register --url http://192.168.199.2:5240/MAAS --secret It still didn't show up, so I deleted it and attempted to rejoin again. No luck. I believe that this issue may have been caused by renaming the instance AFTER installing MAAS initially. Purging and reinstalling to see if I can recreate the issue a second time without injecting any more variables into the scenario.