tripleo

octavia test_basic_http_traffic on c9 times out as it asks for password while connecting to the vm

Bug #1958531 reported by Ananya Banerjee
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo yoga-2

Bug Description

octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_basic_http_traffic on centos stream 9 times out as it asks for password while connecting to the vm.

This should not be the proper behavior. On tempest tests, the ssh should be injected in the vm, and this should be accessed without the user interact.

The test is hanging exactly because of this, if you run the tempest manually, at some point, after the vm is on ACTIVE status, and the test tries to ssh to the vm, it ask for the password.

We tried to add into tempest.conf the image_ssh_password to see if we can workaround this, but no success.

We are pretty sure this is related to octavia test itself, because we teste the test_network_basic_ops, that do exactly the same (create a vm, and ssh using ssh key) and it works.

Bellow is the tempest.conf section for octavia that is being used:

```
[load_balancer]
test_server_path = "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/contrib/test_server/test_server.bin"
test_with_ipv6 = False
enable_security_groups = True
admin_role = admin
RBAC_test_type = owner_or_admin
member_role = member
region = regionOne
enabled_provider_drivers = amphora:The Octavia Amphora driver.,octavia:Deprecated alias of the Octavia Amphora driver.,ovn:Octavia OVN driver.
```

Right now, we are not sure if this requires other configuration on load_balancer section or in the code of the test itself.

We were able to access the cirros VM manually, and we can confirm that the authorized_key is there on .ssh/authorized_keys

We also notice adding a different ssh key in the authorized_keys did not work. so it seems to be how the vm is being created (again, the basic_network_scenario_ops worked fine).

Looks like a problem with how the VM boots up in octavia.

Ananya Banerjee (frenzyfriday)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → yoga-2
tags: added: promotion-blocker
Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

There's an incompatibility between Centos 9 stream ssh client and cirros ssh server, the scp connection initiated in the octavia-tempest-plugin fails (but the RemoteClient class works).

A workaround would be to extend the PubkeyAcceptedKeyTypes for scp with:

-o PubkeyAcceptedKeyTypes=+rsa-sha2-256,rsa-sha2-512

Revision history for this message
Arx Cruz (arxcruz) wrote :

So, adding the -o PubkeyAcceptedKeyTypes=+rsa-sha2-256,rsa-sha2-512 avoid the job to be stucked in the prompt password as Grebory mention, however, since scp now uses sftp protocol by default [1] the test still fails with cannot find /usr/libexec/sftp-server. This is a configuration that needs to be done on cirros side. From what I notice, the dropbear ssh server is not compiled with sftp support (then the /usr/libexec/sftp-server cannot be found).

According to [1] you can still use the old scp protocol passing the -O option, however, I don't know how to do it on the tempest side, I'm gonna need more research on that.

Also, this config must be set either on /etc/ssh/ssh_config or in /root/.ssh/config, and I'm not sure how this would affect other tests, or other playbooks that is running.

1 - https://groups.google.com/g/opensshunixdev/c/LeaacTEj0j0?pli=1

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

I'm working on a fix for octavia-tempest-plugin: https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/816369

It fixes the key type issue and the sftp/scp issue.

The job is not yet passing in the CI because of another issue with c9s

Revision history for this message
Arx Cruz (arxcruz) wrote :

Thanks, I didn't saw your comment here and worked in a different approach https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/826633

Revision history for this message
Marios Andreou (marios-b) wrote :

can you please always include links to the failing tests - i cannot see any examples in the description

Revision history for this message
Ananya Banerjee (frenzyfriday) wrote :
Download full text (11.8 KiB)

Some background:

periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master was timing out without any obvious errors during the tempest tests. The tempest run log showed something like this [1]
After holding the node and running the octavia tests one by one it was found that octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_basic_http_traffic asks for password while connecting to the VM.

[1]
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_add_protocol_to_identity_provider [2.259778s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_add_protocol_to_identity_provider_unknown_mapping [1.402297s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_get_protocol_from_identity_provider [1.803398s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_create [1.058196s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_create_with_enabled_true [0.910964s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_create_with_remote_ids [1.278733s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_get [1.041299s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_list [2.095542s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_identity_provider_update [0.636193s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_list_protocols_from_identity_provider [0.800037s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_update_mapping_from_identity_provider_protocol [0.694822s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest.test_update_protocol_from_identity_provider_unknown_mapping [1.010456s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_authorize_request_token [0.514005s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_create_access_token [0.317166s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_create_and_show_consumer [0.108660s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_create_request_token [0.119455s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_delete_consumer [0.206763s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_list_access_tokens [0.752549s] ... ok
{0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens....

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :

I tested on a a downstream run "periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master" with these two depends-ons: https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/816369 and https://review.opendev.org/c/openstack/octavia/+/816370

The job passed.

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.