MAAS

[2.0 RC2] Static IP address are allowed to be created in a dynamic range

Bug #1604461 reported by Larry Michel on 2016-07-19

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Released	Critical	Mike Pontillo
	2.0	Fix Released	Critical	Mike Pontillo	MAAS 2.0.0

Bug Description

The regiond logs full of these exception below. It's not clear as to how we got to this point, but the IP address in question is from the dynamic range.

i/2.0/users/?op=whoami HTTP/1.1" 200 80 "-" "Go-http-client/1.1"
2016-07-17 08:15:47 [-] Unhandled failure in updating lease.
Traceback (most recent call last):
   File "/usr/lib/python3.5/threading.py", line 862, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 904, in worker
     return target()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
--- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 937, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 205, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 602, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 433, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.5/contextlib.py", line 30, in inner
     return func(*args, **kwds)
   File "/usr/lib/python3/dist-packages/maasserver/rpc/leases.py", line 134, in update_lease
     lease_time=lease_time, created=created, updated=created)
   File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 348, in create
     obj.save(force_insert=True, using=self.db)
   File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 29, in save
     return super(CleanSave, self).save(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 64, in save
     return super(TimestampedModel, self).save(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 734, in save
     force_update=force_update, update_fields=update_fields)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 762, in save_base
     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 846, in _save_table
     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 885, in _do_insert
     using=using, raw=raw)
   File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 920, in _insert
     return query.get_compiler(using=using).execute_sql(return_id)
   File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
     cursor.execute(sql, params)
   File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
     return self.cursor.execute(sql, params)
   File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
     six.reraise(dj_exc_type, dj_exc_value, traceback)
   File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
     raise value.with_traceback(tb)
   File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
     return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key"
DETAIL: Key (ip)=(**.***.**.**) already exists.

2016-07-17 08:15:50 [-] 127.0.0.1 - - [17/Jul/2016:08:15:49 +0000] "GET /MAAS/api/2.0/version/ HTTP/1.1" 200 151 "-" "Go-http-client/1.1"

ubuntu@maas2-production:/var/log/maas$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==================================-============-=================================================
ii maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)

I am attaching the logs and dhcp lease files.

Tags:

Related branches

lp:~mpontillo/maas/prevent-static-address-in-dynamic-range--bug-1604461

Merged into lp:~maas-committers/maas/trunk at revision 5220

Andres Rodriguez (community): Approve on 2016-07-29

lp:~mpontillo/maas/prevent-static-address-in-dynamic-range--bug-1604461--2.0

Merged into lp:maas/2.0 at revision 5183

Mike Pontillo (community): Approve on 2016-07-30

Revision history for this message

Larry Michel (lmic) wrote on 2016-07-19:

logs.tar.gz Edit (11.8 MiB, application/x-tar)

Revision history for this message

Larry Michel (lmic) wrote on 2016-07-19:

Seeing more duplicate and they all seem to be in the dynamic range.

Revision history for this message

Blake Rouse (blake-rouse) wrote on 2016-07-19:

The issue is that you reserved static IP addresses in the dynamic range. This should not be allowed, a dynamic range should also not be allowed to be created over any static IP addresses.

summary:	- [2.0 RC2] regiond log errors: duplicate key value violates unique - constraint "maasserver_staticipaddress_ip_key" DETAIL: Key - (ip)=(...*) already exist + [2.0 RC2] Static IP address are allowed to be created in a dynamic + range, and the reverse is true
summary:	[2.0 RC2] Static IP address are allowed to be created in a dynamic - range, and the reverse is true + range and the reverse is true
Changed in maas:
status:	New → Triaged
importance:	Undecided → Critical
milestone:	none → 2.0.0

Revision history for this message

Mike Pontillo (mpontillo) wrote on 2016-07-28: Re: [2.0 RC2] Static IP address are allowed to be created in a dynamic range and the reverse is true

There is already code that checks if an IP address has been allocated inside the dynamic range (and if so, prevents creation of the range).

Currently, dynamic ranges cannot overlap other ranges, or reserved IP addresses. Reserved ranges can overlap reserved IP addresses, but not other ranges.

So I think the only thing to fix here is to verify that a user-reserved IP address is not within a dynamic range when reserving it.

(As an aside, if you wish to reserve an IP addresses inside a dynamic range, create a device first so that MAAS can add a DHCP mapping for the MAC to the reserved IP.)

summary:

[2.0 RC2] Static IP address are allowed to be created in a dynamic
- range and the reverse is true
+ range

MAAS Lander (maas-lander) on 2016-07-29

Changed in maas:
status:	Triaged → Fix Committed

Revision history for this message

Trevor G Francis (trevor-francis) wrote on 2016-08-09:

Any chance of escalating this into Beta 8 and releasing it. It basically renders MAAS unable to commission nodes.

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2016-08-09: Re: [Bug 1604461] Re: [2.0 RC2] Static IP address are allowed to be created in a dynamic range

Hi Trevor,

Why are you still in beta8? MAAS 2.0 rc3 has been released a few weeks
ago...

PPA:Maas/next

On Monday, August 8, 2016, Trevor G Francis <trevor.francis@46labs.com>
wrote:

> Any chance of escalating this into Beta 8 and releasing it. It basically
> renders MAAS unable to commission nodes.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1604461
>
> Title:
> [2.0 RC2] Static IP address are allowed to be created in a dynamic
> range
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1604461/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Revision history for this message

Trevor G Francis (trevor-francis) wrote on 2016-08-11:

Download full text (7.0 KiB)

Sorry about that. Updated today....issue still exists:

MAAS Version 2.0.0 (rc4+bzr5187)

Sorry about that. Updated today....issue still exists:

MAAS Version 2.0.0 (rc4+bzr5187)

Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: 2016-08-11 00:38:03 [-] Unhandled failure in updating lease.
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011Traceback (most recent call last):
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3.5/threading.py", line 862, in run
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    self._target(*self._args, **self._kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 904, in worker
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return target()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    task()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    task()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011--- <exception caught here> ---
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    result = inContext.theWork()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return self.currentContext().callWithContext(ctx, func, *args, **kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func(*args,**kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 937, in callInContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 205, in wrapper
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 602, in call_within_transaction
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func_outside_txn(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 433, in retrier
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return func(*args, **kwds)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/maasserver/rpc/leases.py", line 134, in update_lease
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    lease_time=lease_time, created=created, updated=created)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return getattr(self.get_queryset(), name)(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 348, in create
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    obj.save(force_insert=True, using=self.db)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 29, in save
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return super(CleanSave, self).save(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 64, in save
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return super(TimestampedModel, self).save(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 734, in save
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    force_update=force_update, update_fields=update_fields)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 762, in save_base
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 846, in _save_table
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 885, in _do_insert
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    using=using, raw=raw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return getattr(self.get_queryset(), name)(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 920, in _insert
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return query.get_compiler(using=using).execute_sql(return_id)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    cursor.execute(sql, params)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return self.cursor.execute(sql, params)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    six.reraise(dj_exc_type, dj_exc_value, traceback)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    raise value.with_traceback(tb)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011    return self.cursor.execute(sql, params)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key"
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011DETAIL:  Key (ip)=(172.16.101.38) already exists.
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2016-08-11:

Hi Trevor,

The bugfix will prevent any *future* creation of IP reservations inside a dynamic rnage, but it won't clean up any reservation already made. You'll need to do that by unreserving the IP address inside the dynamic range.

Hope this helps.

Revision history for this message

Trevor G Francis (trevor-francis) wrote on 2016-08-11:

Can you suggest how to do that?

Andres Rodriguez (andreserl) on 2016-08-22