Comment 16 for bug 1641615

after init is done and vars can be checked
b virnettlssessiontest.c:95
Then it initializes server and client context without checks intentionally
to detect problems via the TLS session validation stage.
 virNetTLSContextNewServer
 virNetTLSContextNewClient
Both work (wildcard is used on Server Context creation)
Then it creates sessions via
  virNetTLSSessionNew
also working.
Callbacks on the socket pair are registered (testWrite/testRead)
  virNetTLSSessionSetIOCallbacks
Then it is looping until a handshake completes or fails.
  virNetTLSSessionHandshake
The handshake completes and then the validation is called for server and client
  virNetTLSContextCheckCertificate

Do note that as outlined before the "libvirt: XML-RPC error : authentication failed:
Failed to verify peer's certificate" can be ok as it is also checking for "expected to fail" certificates.

Actually a lot of good vir DEBUG/WARN in there - set env accordingly.
LIBVIRT_DEBUG=1 VIR_TEST_DEBUG=1 VIR_TEST_VERBOSE=1 ./.libs/lt-virnettlssessiontest

  debug : virNetTLSSessionHandshake:1342 : Handshake is complete
  debug : virNetTLSContextValidCertificate:1063 : Peer DN is CN=libvirt,C=UK
  debug : virNetTLSContextCheckCertDNWhitelist:387 : Failed whitelist check for client DN 'CN=libvirt,C=UK'
  info : virNetTLSContextValidCertificate:1105 : RPC_TLS_CONTEXT_SESSION_DENY: ctxt=0x55fe2c5673b0 sess=0x55fe2c572d70 dname=CN=libvirt,C=UK
  warning : virNetTLSContextCheckCertificate:1125 : Certificate check failed Client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list). Use 'certtool -i --infile clientcert.pem' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option.
  warning : testTLSSessionInit:192 : Unexpected server cert check fail

The actual check is made in virNetTLSContextValidCertificate which does various gnutls calls

next go for a good and bad case check with upstream code - as it seems likely this needs to be fixed there.