Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic
Charmed Openstack deployment with HW Offload over Jammy series will look ok until Vault initialization phase, then after initializing Vault all DB-related apps will end up in block/error state over "Failed to connect to MYSQL".
Connectivity testing between DB containers located on different nodes will show there is unexplained sporadic packet loss preventing proper communication between the DB related apps.
This will happen when the following conditions are met:
1. Control plane (oam, internal spaces) is configured as vlan interfaces on the same OVS bridge used for data plane (over high speed NIC with HW Offload capabilities).
2. OVS was set with HW offload=true (will happen by OVN chrams after Vault initialization)
3. NIC was not yet set to "switchdev" mode (netplan file will be created by OVN chrams after Vault initialization, however will take affect only after node is rebooted)
The root cause is the following missing kernel patch:
https://patchwork<email address hidden>/
To reproduce:
Deploy charmed openstack with HW offload while using control plane on the high speed NIC OVS bridge. Before initializing Vault login to one of the innoDB instances and ping the other 2 instances - all ok. Manually enable OVS HW Offload, ping will become inconsistent.
Workaround:
After the deployment bring-up phase, BEFORE enabling Vault, login to the nodes and manually create 150-charm-ovn.yaml (example below). Then reboot one node after another. When nodes recover proceed with Vault initialization to complete the deployment.
#root@node3:/home/ubuntu# cat /etc/netplan/150-charm-ovn.yaml
###############################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
# Config managed by ovn-chassis charm
###############################################################################
network:
version: 2
ethernets:
ens1f0:
virtual-function-count: 8
embedded-switch-mode: switchdev
delay-virtual-functions-rebind: true
ens1f1:
virtual-function-count: 8
embedded-switch-mode: switchdev
delay-virtual-functions-rebind: true
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jul 4 10:46 seq
crw-rw---- 1 root audio 116, 33 Jul 4 10:46 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-40-generic root=UUID=db1801a9-daa1-4386-b2ec-c65a40bc5dd3 ro intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
ProcVersionSignature: Ubuntu 5.15.0-40.43-generic 5.15.35
RelatedPackageVersions:
linux-restricted-modules-5.15.0-40-generic N/A
linux-backports-modules-5.15.0-40-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu3.2
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-40-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 03/05/2015
dmi.bios.release: 1.32
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 2.53
dmi.modalias: dmi:bvnHP:bvrP89:bd03/05/2015:br1.32:efr2.53:svnHP:pnProLiantDL360Gen9:pvr:cvnHP:ct23:cvr:sku755258-B21:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.product.sku: 755258-B21
dmi.sys.vendor: HP
apport information