autofs: Assertion 'set_remove(iterator->links, link) == link' failed at src/shared/userdb.c:314, function userdb_on_query_reply(). Aborting.

Instead of using autofs, have you tried to simply use systemd .automount units which should be able to achieve similar behaviour?

Furthermore, .automount units might be better, as they also could be use not only as _system_ amountmount units but as _user_ systemd .automount users.

See http://manpages.ubuntu.com/manpages/focal/en/man5/systemd.automount.5.html

No, I didn't try to use systemd .automount units. Unfortunately it doesn't fit to me, because my AutoFS Maps are served by Active Directory (Samba AD on Debian 10). Then Linux clients receive these maps from AD and mount them dynamically (I don't configure mounts on each particular PC). As I understand systemd .automount units doesn't allow to get mounts list from Active Directory.

There's some manual that describes configuration of AutoFS Maps in AD (very similar to my configuration):
https://care.qumulo.com/hc/en-us/articles/115014470007-Active-Directory-AutoFS-maps-to-AD-bound-Linux-clients-with-SSSD

Just FYI, it works correctly on Ubuntu 18.04 and Fedora 32.

From systemd 245 release notes (https://lwn.net/Articles/814068/):

----
  * A new component "userdb" has been added, along with a small daemon
          "systemd-userdb.service" and a client tool "userdbctl". The framework
          allows defining rich user and group records in a JSON format,
          extending on the classic "struct passwd" and "struct group"
          structures. Various components in systemd have been updated to
          process records in this format, including systemd-logind and
          pam-systemd. The user records are intended to be extensible, and
          allow setting various resource management, security and runtime
          parameters that shall be applied to processes and sessions of the
          user as they log in. This facility is intended to allow associating
          such metadata directly with user/group records so that they can be
          produced, extended and consumed in unified form. We hope that
          eventually frameworks such as sssd will generate records this way, so
          that for the first time resource management and various other
          per-user settings can be configured in LDAP directories and then
          provided to systemd (specifically to systemd-logind and pam-system)
          to apply on login. For further details see:

          https://systemd.io/USER_RECORD
          https://systemd.io/GROUP_RECORD
          https://systemd.io/USER_GROUP_API
----

and yet we don't have userdbctl tool or the daemon

https://www.freedesktop.org/software/systemd/man/userdbctl.html

looks like an ongoing effort of unifying user/group information coming from
pam-systemd to logind management scheme within systemd.

I believe making all information coming from pam-systemd to logind available
through this varlink interface is what is causing the issue and where the problem
relies.

----

Nevertheless...

Error is coming from the userdb codeset, from the assertion:

        assert_se(set_remove(iterator->links, link) == link);

when userdb code is being called by the varlink protocol.

Many subsystems within systemd now have an embedded varlink server to provide
IPC through simple json protocol. The journal daemon creates a varlink server on its
own through systemd-journald -> server_init -> server_open_varlink() ->
varlink_server_listen_fd() being one example.

The execution path for this error is either coming from:

(1)

process_connection() -> varlink_process() -> varlink_dispatch_reply() -> reply_callback()

and the reply_callback is a pointer to userdb_on_query_reply(), since this callback is set with varlink_bind_reply().

if (IN_SET(v->state, VARLINK_AWAITING_REPLY, VARLINK_AWAITING_REPLY_MORE)) {
    varlink_set_state(v, VARLINK_PROCESSING_REPLY);

if (v->reply_callback)
    r = v->reply_callback(v, parameters, error, flags, v->userdata)

OR

(2) from an error coming from:

varlink_dispatch_disconnect()
varlink_dispatch_method()
varlink_dispatch_reply()
varlink_dispatch_timeout()

all of them calling varlink_dispatch_local_error().

These errors come from varlink_process() main logic, processing the varlink protocol.

- A timeout in connection would trigger varlink_dispat...

With that said... perhaps if you can reproduce this easily you could reproduce with:

systemd.log_level=debug systemd.log_target=syslog systemd.dump_core=true

added to your kernel cmdline, and provide me related debug text.

And also, updating your systemd to latest in Focal:

245.4-4ubuntu3.2

and providing me the core dump from the user session abort() generated by:

"Assertion 'set_remove(iterator->links, link) == link' failed at src/shared/userdb.c:314, function userdb_on_query_reply(). Aborting.

Aborted (core dumped)"

That would help me out trying to identify the issue.

I'll continue to reply here as you continue to provide me feedback about this issue...

Thank you.

-rafaeldtinoco

Thank you a lot for this investigation, I'll try to reproduce with these parameters and with new version on this week.

This error has gone with the latest updates, but, unfortunately, I still have the same transient issues with autofs. I see some other strange things in logs and will continue investigation on my side. I think this ticket can be closed (I will open a new ticket if any). Thank you a lot.

Hello Michael, thanks for the followup and for filing the bug in the first place.

For the moment I'm changing the status of this report to Incomplete across the packages/series it targets. If this specific issue can't be reproduced anymore please set the statuses to Invalid (I'd prefer it to Fix Released as we didn't identify what actually fixed it), and go ahead filing a new bug for the remaining issues you're facing. On the other hand if you still think this should be investigated please comment back with your findings, change the bug status back to New and we'll look at it again. Thanks!