MAAS fires 5 tasks to write_dhcp_config when only one nodegroup change was made

Bug #1324944 reported by Diogo Matsubara
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Raphaël Badin
1.5
Fix Released
Critical
Raphaël Badin
maas (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Fix Released
Undecided
Unassigned

Bug Description

It was observed in http://d-jenkins.ubuntu-ci:8080/view/MAAS/job/utopic-adt-maas/45/artifact/results/artifacts/maas-logs/var/log/maas/celery.log/*view*/ that MAAS triggers 5 write_dhcp_config tasks when only one nodegroup change was requested through the MAAS CLI (see the tests run by that jenkins job lines 407 to 444: http://bazaar.launchpad.net/~maas-maintainers/maas/qa-lab-tests/view/head:/maas-integration.py)

This coupled to the fact that celery is not executing the tasks FIFO (see bug 1324943) seem to have a caused a race condition where the dhcp config done through the CLI is overwritten by a previous tasks.

[Impact]
 * Remove possible race condition that might cause changes to dhcpd.conf to be lost when stopping dhcpd

[Testcase]
 * Install maas and run included test test_stop_dhcp_server_sends_command_and_writes_empty_config and test_set_up_dhcp_region from integration tests and check celery.log has only one write_dhcp_config task triggered

[Regression Potential]
 * Minimal -- simply write blank config instead of invalid configs and only once

Related branches

Revision history for this message
Raphaël Badin (rvb) wrote :

Part of the problem here is that MAAS generates a DHCP config even when there is no interface configured. Worse: when there is no interface configured, the config it generates it invalid.

Changed in maas:
importance: Undecided → High
status: New → Triaged
Changed in maas:
milestone: none → 1.6.0
Revision history for this message
Raphaël Badin (rvb) wrote :

Actually, this consequence of this bug was MAAS generating an invalid DHCP config that was causing the DHCP server to crash so I'm marking this bug as Critical and will backport the fix to 1.5.

Changed in maas:
importance: High → Critical
Changed in maas:
milestone: 1.6.0 → none
Raphaël Badin (rvb)
Changed in maas:
assignee: nobody → Raphaël Badin (rvb)
status: Triaged → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
Revision history for this message
Greg Lutostanski (lutostag) wrote :

branch which fixes this and other 1.5.2 release bugs for SRU is linked to bug #1325038

description: updated
Changed in maas (Ubuntu):
status: New → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Diogo, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.2+bzr2282-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in maas (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Diogo Matsubara (matsubara) wrote :
tags: added: verification-done
removed: verification-needed
Revision history for this message
Stéphane Graber (stgraber) wrote :

Hello Diogo, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.2+bzr2282-0ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Greg Lutostanski (lutostag) wrote :

Verified in maas lab with tests as shown by Diogo, also ran the two integration tests manually, and both passed.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 1.5.2+bzr2282-0ubuntu0.2

---------------
maas (1.5.2+bzr2282-0ubuntu0.2) trusty-proposed; urgency=medium

  * debian/control:
   - Add missing dependency in maas-cluster-controller for grub-common
     (LP: #1328231)
   - Move dependency from maas-cluster-controller to maas-provisioningserver
     for python-seamicroclient (LP: #1332532)

maas (1.5.2+bzr2282-0ubuntu0.1) trusty-proposed; urgency=medium

  * New upstream release:
    - Remove workaround for fixed Django bug 1311433 (LP: #1311433)
    - Ensure that validation errors are returned when adding a node over
      the API and its cluster controller is not contactable. (LP: #1305061)
    - Hardware enablement support for PowerKVM (LP: #1325038)
    - Shorten the time taken for a cluster to initially connect to the region
      via RPC to around 2 seconds (LP: #1317682)
    - Faster DHCP leases parser (LP: #1305102)
    - Documentation fixed explaining how to enable an ephemeral backdoor
      (LP: #1321696)
    - Use probe-and-enlist-hardware to enlist all virtual machine inside
      a libvirt machine, allow password qemu+ssh connections.
      (LP: #1315155, LP: #1315157)
    - Rename ppc64el boot loader to PowerKVM (LP: #1315154)
    - Fix NodeForm's is_valid() method so that it uses Django's way of setting
      errors on forms instead of putting text in self.errors['architecture']
      (LP: #1301465)
    - Change BootMethods to return their own IReader per-request, update method
      names to reflect new usage. (LP: #1315154)
    - Return early and stop the DHCP server when the list of managed interfaces
      of the nodegroup is empty. (LP: #1324944)
    - Fix invalid attribute references in the VirshSSH class. Added more test
      for the VirshSSH class. (LP: #1324966)
  * debian/control:
    - Add missing dependency in maas-cluster-controller for python-pexpect
      (LP: #1322151)
 -- Greg Lutostanski <email address hidden> Fri, 20 Jun 2014 10:10:47 -0500

Changed in maas (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Stéphane Graber (stgraber) wrote : Update Released

The verification of the Stable Release Update for maas has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Chris J Arges (arges)
Changed in maas (Ubuntu Trusty):
status: Fix Released → In Progress
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Diogo, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.4+bzr2294-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in maas (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

This has been tested and confirmed that the issue is fixed.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 1.5.4+bzr2294-0ubuntu1.1

---------------
maas (1.5.4+bzr2294-0ubuntu1.1) trusty-proposed; urgency=medium

  * Add hardware enablement for armhf/keystone (LP: #1350103)
 -- Greg Lutostanski <email address hidden> Thu, 18 Sep 2014 16:43:56 -0500

Changed in maas (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.