slaac no longer works on IPv6 tenant subnets

Bug #1886116 reported by Michael Johnson
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Brian Haley

Bug Description

Nova instances no longer get an IPv6 address using slaac on tenant subnets.

Using a standard devstack install with "SERVICE_IP_VERSION="6"" added, master (Victoria).

[ml2]
tenant_network_types = vxlan
extension_drivers = port_security
mechanism_drivers = openvswitch,linuxbridge

network:
+---------------------------+--------------------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2020-07-02T22:55:51Z |
| description | |
| dns_domain | None |
| id | e8258754-6a0b-40ea-abf6-c55b39845f62 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| location | cloud='', project.domain_id='default', |
| | project.domain_name=, |
| | project.id='08c84a34e4c34dacb3abbfe840edf6e3', |
| | project.name='admin', region_name='RegionOne', |
| | zone= |
| mtu | 1450 |
| name | lb-mgmt-net |
| port_security_enabled | True |
| project_id | 08c84a34e4c34dacb3abbfe840edf6e3 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 2 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 2f17a970-09b1-410d-89de-c75b1e5f6eef |
| tags | |
| updated_at | 2020-07-02T22:55:52Z |
+---------------------------+--------------------------------------------------+

Subnet:
+----------------------+-------------------------------------------------------+
| Field | Value |
+----------------------+-------------------------------------------------------+
| allocation_pools | fd00:0:0:42::2-fd00::42:ffff:ffff:ffff:ffff |
| cidr | fd00:0:0:42::/64 |
| created_at | 2020-07-02T22:55:52Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | fd00:0:0:42:: |
| host_routes | |
| id | 2f17a970-09b1-410d-89de-c75b1e5f6eef |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| location | cloud='', project.domain_id='default', |
| | project.domain_name=, |
| | project.id='08c84a34e4c34dacb3abbfe840edf6e3', |
| | project.name='admin', region_name='RegionOne', zone= |
| name | lb-mgmt-subnet |
| network_id | e8258754-6a0b-40ea-abf6-c55b39845f62 |
| prefix_length | None |
| project_id | 08c84a34e4c34dacb3abbfe840edf6e3 |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2020-07-02T22:55:52Z |
+----------------------+-------------------------------------------------------+

Security group:
+-----------------+------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------+
| created_at | 2020-07-02T22:55:53Z |
| description | lb-mgmt-sec-grp |
| id | e1a03546-bb32-4102-bf76-bc946e059a45 |
| location | cloud='', project.domain_id='default', |
| | project.domain_name=, |
| | project.id='08c84a34e4c34dacb3abbfe840edf6e3', |
| | project.name='admin', region_name='RegionOne', zone= |
| name | lb-mgmt-sec-grp |
| project_id | 08c84a34e4c34dacb3abbfe840edf6e3 |
| revision_number | 4 |
| rules | created_at='2020-07-02T22:55:53Z', direction='egress', |
| | ethertype='IPv4', |
| | id='806d16f3-b2e4-482b-aea2-45f5284f879d', |
| | updated_at='2020-07-02T22:55:53Z' |
| | created_at='2020-07-02T22:55:54Z', direction='ingress', |
| | ethertype='IPv6', |
| | id='c1fa3d4f-702c-4170-8815-aa75cda902f1', |
| | port_range_max='22', port_range_min='22', protocol='tcp', |
| | remote_ip_prefix='::/0', updated_at='2020-07-02T22:55:54Z' |
| | created_at='2020-07-02T22:55:54Z', direction='ingress', |
| | ethertype='IPv6', |
| | id='d9e6ea44-dcd1-41f1-936f-a757d4e10a7d', |
| | protocol='ipv6-icmp', remote_ip_prefix='::/0', |
| | updated_at='2020-07-02T22:55:54Z' |
| | created_at='2020-07-02T22:55:55Z', direction='ingress', |
| | ethertype='IPv6', |
| | id='e47a799d-bbda-403b-aeb4-dc307ce7fa69', |
| | port_range_max='9443', port_range_min='9443', |
| | protocol='tcp', remote_ip_prefix='::/0', |
| | updated_at='2020-07-02T22:55:55Z' |
| | created_at='2020-07-02T22:55:53Z', direction='egress', |
| | ethertype='IPv6', id='fddd23d8-bdbe-420b-83cc- |
| | bad470e3eeb8', updated_at='2020-07-02T22:55:53Z' |
| stateful | True |
| tags | [] |
| updated_at | 2020-07-02T22:55:55Z |
+-----------------+------------------------------------------------------------+

Boot a nova instance on this network/subnet:
+-------------------------------------+----------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-07-03T00:12:31.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | lb-mgmt- |
| | net=fd00::42:f816:3eff:fe5b:b7b3 |
| config_drive | True |
| created | 2020-07-03T00:12:15Z |
| flavor | m1.amphora |
| | (8e2df9bb-6268-44f0-b4c2-45d57c59240a) |
| hostId | 4569b0d1a044a24e8ea70c8b6715181c1930bd |
| | acf551c6dd13220daa |
| id | 418b4304-616e-42d3-9cb9-6a965e3a5fd3 |
| image | amphora-x64-haproxy |
| | (4d1800d5-03b9-46af-a69f-99987bd829a5) |
| key_name | octavia_ssh_key |
| name | amphora-1f931e78-0961-4a26-9f22-da0a72 |
| | 67fff6 |
| progress | 0 |
| project_id | 08c84a34e4c34dacb3abbfe840edf6e3 |
| properties | |
| security_groups | name='lb-mgmt-sec-grp' |
| status | ACTIVE |
| updated | 2020-07-03T00:12:32Z |
| user_id | 4316f8b3e21d4dee9f9ce6a7deaad234 |
| volumes_attached | |
+-------------------------------------+----------------------------------------+

This all looks ok, but the instance will never receive an IP address on the fd00::42: subnet like it has in the past.

The dnsmasq process for this subnet:
nobody 30554 1 0 15:55 ? 00:00:00 dnsmasq --no-hosts --pid-file=/opt/stack/data/neutron/dhcp/e8258754-6a0b-40ea-abf6-c55b39845f62/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/e8258754-6a0b-40ea-abf6-c55b39845f62/host --addn-hosts=/opt/stack/data/neutron/dhcp/e8258754-6a0b-40ea-abf6-c55b39845f62/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/e8258754-6a0b-40ea-abf6-c55b39845f62/opts --dhcp-leasefile=/opt/stack/data/neutron/dhcp/e8258754-6a0b-40ea-abf6-c55b39845f62/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=0 --conf-file= --domain=openstacklocal

The only running radvd process has the following config file:

interface qr-1518a0f2-75
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;

   AdvLinkMTU 1450;

   prefix fdf2:3712:3235::/64
   {
        AdvOnLink on;
        AdvAutonomous on;
   };

};

NetNS information:
ip netns exec qdhcp-e8258754-6a0b-40ea-abf6-c55b39845f62 bash

root@devstack:~/devstack# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
13: tap610b5474-94: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:bf:e4:a6 brd ff:ff:ff:ff:ff:ff
    inet6 fd00::42:f816:3eff:febf:e4a6/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:febf:e4a6/64 scope link
       valid_lft forever preferred_lft forever

tcpdump:
 tcpdump -nli tap610b5474-94
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap610b5474-94, link-type EN10MB (Ethernet), capture size 262144 bytes
17:31:21.407531 IP6 fd00::42:f816:3eff:fe2a:dc4d > ff02::1:ff5b:b7b3: ICMP6, neighbor solicitation, who has fd00::42:f816:3eff:fe5b:b7b3, length 32
17:31:23.435505 IP6 fd00::42:f816:3eff:fe2a:dc4d > ff02::1:ff5b:b7b3: ICMP6, neighbor solicitation, who has fd00::42:f816:3eff:fe5b:b7b3, length 32
17:31:24.451503 IP6 fd00::42:f816:3eff:fe2a:dc4d > ff02::1:ff5b:b7b3: ICMP6, neighbor solicitation, who has fd00::42:f816:3eff:fe5b:b7b3, length 32
17:31:25.471526 IP6 fd00::42:f816:3eff:fe2a:dc4d > ff02::1:ff5b:b7b3: ICMP6, neighbor solicitation, who has fd00::42:f816:3eff:fe5b:b7b3, length 32

It appears to me that there is nothing on the network responding to the neighbor solicitations for SLAAC.

Tags: doc ipv6
Revision history for this message
Michael Johnson (johnsom) wrote :

Adding a router to the subnet starts a radvd and the nova instance gets an IP address.

I guess something changed that subnets without a router don't have slaac support? Isn't dnsmasq supposed to be providing slaac (I would have expected to see ra-only in the dnsmasq command line)? Are there no tests for slaac subnets?

Revision history for this message
Lajos Katona (lajos-katona) wrote :

Hi, based on this https://docs.openstack.org/neutron/latest/admin/config-ipv6.html#using-slaac-for-addressing, I assume that router is necessary for slaac, though haven't checked it historically.

Revision history for this message
Brian Haley (brian-haley) wrote :

Michael - so I wasn't crazy, RAs were being done by dnsmasq on isolated networks, but the change was reverted because there were some CentOS systems running an ancient dnsmasq that didn't support it.

https://review.opendev.org/#/c/393007/
https://review.opendev.org/#/c/401985/

I think we should re-propose a version of that first one, but perhaps just for isolated subnets, I think it was doing for all.

And the commit message even says "Implementing this now so Octavia can use an isolated IPv6
network.", doh!

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
tags: added: ipv6
Revision history for this message
Brian Haley (brian-haley) wrote :

Jens had a comment in the bug were the IPv6 SLAAC code was added to the DHCP agent:

https://bugs.launchpad.net/neutron/+bug/1498987/comments/14

"A dhcp agent is no router, so IMO it should not be sending router advertisements. If people want slaac to work on an isolated network, why can't they just attach a router without a gateway? We could then fix the l3 code to set the router lifetime to zero if the router has no gateway set (if that should not yet be the case)."

I think this is a good point - IPv6 configuration in general doesn't work very well without the presence of a router. Having dnsmasq send RAs is kind-of a hack in the neutron case, and even the code that originally added this support made it so that two entities (radvd and dnsmasq) would be sending RAs in most cases. This is undesirable, so I think the best option is to document this better in the admin guide, and also make sure the router lifetime is set to zero if there is no external gateway configured.

Changed in neutron:
importance: High → Low
Revision history for this message
Brian Haley (brian-haley) wrote :

And just to add one more thing, I don't know why this was working previously for Octavia, as there was nothing sending an RA in the IPv6-only case that should have caused SLAAC to succeed.

tags: added: doc
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/739848

Changed in neutron:
assignee: nobody → Brian Haley (brian-haley)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/739848
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a3ecaf6a1078ae1e1edbe72db0fd612e826539ad
Submitter: Zuul
Branch: master

commit a3ecaf6a1078ae1e1edbe72db0fd612e826539ad
Author: Brian Haley <email address hidden>
Date: Tue Jul 7 16:29:29 2020 -0400

    Better document router requirements for IPv6

    In order for IPv6 to function correctly for instances, a router
    must be created and added to a subnet. Update the documentation
    to better highlight this as it wasn't clear a router was
    required on an isolated subnet such that Router Advertisements
    messages would be sent.

    Change-Id: I4aca67c98ae77bbc4c130764af5a92515b95443a
    Closes-bug: #1886116

Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.