Fuel for OpenStack

Make default gateway configuration idempotent

Bug #1541309 reported by Aleksandr Didenko
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Aleksandr Didenko
Fuel for OpenStack 10.0
8.0.x
Won't Fix
Medium
Unassigned
Fuel for OpenStack 8.0
Mitaka
Fix Released
High
Aleksandr Didenko
Fuel for OpenStack 9.0

Bug Description

Since implementation of routing via vrouter mgmt/vip (in order to remove public network from non-controllers) and applying a fix for #1444962 we have non-idempotent network configuration process for some roles:
https://review.openstack.org/#/c/176665/23/deployment/puppet/osnailyfacter/modular/astute/tasks.yaml

So if we don't use public network, then for those roles we configure routing via admin network with netconfig.pp task of deployment stage and then re-configure default gateway via vrouter mgmt/vip with configure_default_route.pp task of post-deployment stage.

For role based deployment we can improve this with the following change in "configure_default_route" task:
- role: [primary-mongo, mongo, compute, ceph-osd, cinder, zabbix-server, base-os]
+ role: [primary-mongo, mongo, zabbix-server, base-os]

But for new task based deployment we can try to get rid of configure_default_route completely: start netconfig.pp task on non-controller roles only after virtual_ips.pp task is completed on controller nodes (thus all the VIPs are in place).

Revision history for this message
Aleksandr Didenko (adidenko) wrote :

I guess it's too late to improve this in 8.0 due to HCF, so I've marked it as "won't fix" for 8.0

Aleksandr Didenko (adidenko)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Aleksandr Didenko (adidenko)
Dmitry Pyzhov (dpyzhov)
tags: added: tech-debt
Changed in fuel:
status: New → Confirmed
Aleksandr Didenko (adidenko)
tags: added: team-network
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/299294

Changed in fuel:
status: Confirmed → In Progress
Fuel Devops McRobotson (fuel-devops-robot)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote : Re: Make network configuration idempotent

Technical debt, doesn't affect real users. Removing from Mitaka release.

Revision history for this message
Aleksandr Didenko (adidenko) wrote :

This bug is not tech-debt. This bug is important for LCM because it may lead to network connection failures since two different tasks in our deployment graph configure default route differently. I've updated the bug.

tags: removed: tech-debt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/299294
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7903d3d152503da746d2610bb54a4ab5e57ad6bf
Submitter: Jenkins
Branch: master

commit 7903d3d152503da746d2610bb54a4ab5e57ad6bf
Author: Aleksandr Didenko <email address hidden>
Date: Wed Mar 30 12:30:21 2016 +0200

    Make network configuration idempotent

    Remove configure_default_route task to make network configuration
    idempotent for majority of roles. Only mongo roles still have
    non-idempotent network configuration which is required for old
    role-based deployment.

    Change-Id: I229957b60c85ed94c2d0ba829642dd6e465e9eca
    Partial-bug: #1541309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/317963

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/317963
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fdb541f57de7d8769441ce92c2526807d5694017
Submitter: Jenkins
Branch: stable/mitaka

commit fdb541f57de7d8769441ce92c2526807d5694017
Author: Aleksandr Didenko <email address hidden>
Date: Wed Mar 30 12:30:21 2016 +0200

    Make network configuration idempotent

    Remove configure_default_route task to make network configuration
    idempotent for majority of roles. Only mongo roles still have
    non-idempotent network configuration which is required for old
    role-based deployment.

    Change-Id: I229957b60c85ed94c2d0ba829642dd6e465e9eca
    Partial-bug: #1541309
    (cherry picked from commit I229957b60c85ed94c2d0ba829642dd6e465e9eca)

tags: added: in-stable-mitaka
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
status: In Progress → Fix Committed
Aleksandr Didenko (adidenko)
Changed in fuel:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/320861

Changed in fuel:
assignee: Aleksandr Didenko (adidenko) → Igor Kalnitsky (ikalnitsky)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Vladimir Kuklin (vkuklin)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Igor Kalnitsky (ikalnitsky)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/322047

Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Bulat Gaifullin (bgaifullin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/322047
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=5630e40e610ae72f89541d6f28885fbfac6bc157
Submitter: Jenkins
Branch: master

commit 5630e40e610ae72f89541d6f28885fbfac6bc157
Author: Bulat Gaifullin <email address hidden>
Date: Wed May 25 18:03:48 2016 +0300

    The task attributes substitution is called for all tasks

    The yaql_exp can be used for calculate dependencies of all tasks
    including skipped tasks, so the task attributes traverse should
    be called for all task too.
    Also added check that dependency is not empty object, because
    it is possible when it has been dynamically generated via yaql.

    Partial-Bug: 1541309
    Change-Id: Ibcb786d2a7917d7583433c0b96f6324be4de759b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/320861
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=886ab3087c0eb348e9b88090d69bf462a3e50962
Submitter: Jenkins
Branch: master

commit 886ab3087c0eb348e9b88090d69bf462a3e50962
Author: Igor Kalnitsky <email address hidden>
Date: Wed May 25 12:19:22 2016 +0300

    Allow YAQL expressions in task's dependencies

    Since Fuel 9.0 we have a task-based deployment turned on by default.
    That means our tasks are executing simultaneously on several nodes, and
    old static approach for specifying dependencies doesn't work anymore.

    In order to avoid dependency loops and make possible to rely on
    different tasks based on some condition, we've got to be able to
    calculate `cross-depends` and `cross-depended-by` attributes on fly.

    Fortunately, we already have traversal mechanism that evaluates YAQL
    expressions almost everywhere. So all we need to do is to remove
    validation limitation and allow to upload deployment tasks with YAQL
    expressions in those fields.

    Change-Id: I393dfb06f9e25ecd3ca79fddc84104a8cf993094
    Partial-Bug: #1541309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322753

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322827

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Bulat Gaifullin (bgaifullin) → Aleksandr Didenko (adidenko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/322753
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=8de9381d4877fc208214c1878eca037d3b7594a0
Submitter: Jenkins
Branch: stable/mitaka

commit 8de9381d4877fc208214c1878eca037d3b7594a0
Author: Igor Kalnitsky <email address hidden>
Date: Wed May 25 12:19:22 2016 +0300

    Allow YAQL expressions in task's dependencies

    Since Fuel 9.0 we have a task-based deployment turned on by default.
    That means our tasks are executing simultaneously on several nodes, and
    old static approach for specifying dependencies doesn't work anymore.

    In order to avoid dependency loops and make possible to rely on
    different tasks based on some condition, we've got to be able to
    calculate `cross-depends` and `cross-depended-by` attributes on fly.

    Fortunately, we already have traversal mechanism that evaluates YAQL
    expressions almost everywhere. So all we need to do is to remove
    validation limitation and allow to upload deployment tasks with YAQL
    expressions in those fields.

    (cherry picked from commit 886ab3087c0eb348e9b88090d69bf462a3e50962)

    Change-Id: I393dfb06f9e25ecd3ca79fddc84104a8cf993094
    Partial-Bug: #1541309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/322827
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=0c1fdc6161a505c6bb708134a704c72fc0c0e16f
Submitter: Jenkins
Branch: stable/mitaka

commit 0c1fdc6161a505c6bb708134a704c72fc0c0e16f
Author: Bulat Gaifullin <email address hidden>
Date: Wed May 25 18:03:48 2016 +0300

    The task attributes substitution is called for all tasks

    The yaql_exp can be used for calculate dependencies of all tasks
    including skipped tasks, so the task attributes traverse should
    be called for all task too.
    Also added check that dependency is not empty object, because
    it is possible when it has been dynamically generated via yaql.

    Partial-Bug: 1541309
    Change-Id: Ibcb786d2a7917d7583433c0b96f6324be4de759b
    (cherry picked from commit 5630e40e610ae72f89541d6f28885fbfac6bc157)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/324307

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Aleksandr Didenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/322872
Reason: We'll use this on instead https://review.openstack.org/#/c/324307/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326367

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/324307
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=cbb036fcc3dcb512dbfeb84d06392b9a7ddba12f
Submitter: Jenkins
Branch: master

commit cbb036fcc3dcb512dbfeb84d06392b9a7ddba12f
Author: Aleksandr Didenko <email address hidden>
Date: Mon May 30 16:34:35 2016 +0200

    Make network configuration idempotent

    Remove configure_default_route task to make network configuration
    idempotent for majority of roles. 'Netconfig' task on all
    non-controller (except mongo) nodes will be executed after 'virtual_ips'
    task on controllers. So there's no need to reconfigure default route
    afterwards.

    Only mongo roles still have non-idempotent network configuration during
    the first deployment run, because mongo nodes are deployed before
    controllers in role-based deployment and no VIPs are configured yet.

    Also, instead of having one default gateway in Hiera and then replacing
    it with another one on the fly, let's create a new task that will simply
    put new gateway to Hiera and thus we'll have consistant configuration
    data for future runs of 'netconfig' task even on mongo roles.

    DocImpact

    Change-Id: I7269a6f632f9ab9c88d73be1ef6c04094c49889b
    Closes-bug: #1541309

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/326367
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8d06d9b28fc868ae982cb37a4708892a28d9d47a
Submitter: Jenkins
Branch: stable/mitaka

commit 8d06d9b28fc868ae982cb37a4708892a28d9d47a
Author: Aleksandr Didenko <email address hidden>
Date: Mon May 30 16:34:35 2016 +0200

    Make network configuration idempotent

    Remove configure_default_route task to make network configuration
    idempotent for majority of roles. 'Netconfig' task on all
    non-controller (except mongo) nodes will be executed after 'virtual_ips'
    task on controllers. So there's no need to reconfigure default route
    afterwards.

    Only mongo roles still have non-idempotent network configuration during
    the first deployment run, because mongo nodes are deployed before
    controllers in role-based deployment and no VIPs are configured yet.

    Also, instead of having one default gateway in Hiera and then replacing
    it with another one on the fly, let's create a new task that will simply
    put new gateway to Hiera and thus we'll have consistant configuration
    data for future runs of 'netconfig' task even on mongo roles.

    DocImpact

    Change-Id: I7269a6f632f9ab9c88d73be1ef6c04094c49889b
    Closes-bug: #1541309

Aleksandr Didenko (adidenko)
summary: - Make network configuration idempotent
+ Make default gateway configuration idempotent
Revision history for this message
Alexandr Kostrikov (akostrikov-mirantis) wrote :

Verified on iso 9.0 490:

Default route on mongo node:
http://paste.openstack.org/show/520613/

Default route on controller node:
http://paste.openstack.org/show/520615/

Tasks which has been run(configure_default_route is skipped everywhere except mongo, cluster operational):
http://paste.openstack.org/show/520614/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.