systemd-resolved segfaults

Bug #1809438 reported by OlivierS
60
This bug affects 13 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Invalid
High
Unassigned

Bug Description

I regularly (once almost every hour) get segfaults resported in systemd-resolved. The syslog is not always the same:

Dec 21 06:59:21 marshall kernel: [988042.071003] systemd-resolve[15794]: segfault at e60e8ad514 ip 000000e60e8ad514 sp 00007ffe0376e748 error 14 in systemd-resolved[556f74bcf000+59000]
Dec 21 06:59:21 marshall systemd[1]: systemd-resolved.service: Main process exited, code=dumped, status=11/SEGV
Dec 21 06:59:21 marshall systemd[1]: systemd-resolved.service: Failed with result 'core-dump'.

Dec 21 08:17:47 marshall kernel: [992748.334333] systemd-resolve[16943]: segfault at 2a11d9b0 ip 00007f4810f5ad26 sp 00007fff544db0f0 error 4 in libsystemd-shared-237.so[7f4810e1f000+1b5000]
Dec 21 08:17:47 marshall systemd[1]: systemd-resolved.service: Main process exited, code=dumped, status=11/SEGV
Dec 21 08:17:47 marshall systemd[1]: systemd-resolved.service: Failed with result 'core-dump'.

Dec 21 06:22:45 marshall systemd-resolved[15727]: Assertion 'p->n_ref > 0' failed at ../src/resolve/resolved-dns-packet.c:210, function dns_packet_unref(). Aborting.
Dec 21 06:22:45 marshall systemd[1]: systemd-resolved.service: Main process exited, code=dumped, status=6/ABRT
Dec 21 06:22:45 marshall systemd[1]: systemd-resolved.service: Failed with result 'core-dump'.

Dec 21 06:17:46 marshall systemd-resolved[15662]: Assertion 'DNS_TRANSACTION_IS_LIVE(q->state)' failed at ../src/resolve/resolved-dns-query.c:540, function dns_query_complete(). Aborting.
Dec 21 06:17:46 marshall systemd[1]: systemd-resolved.service: Main process exited, code=dumped, status=6/ABRT
Dec 21 06:17:46 marshall systemd[1]: systemd-resolved.service: Failed with result 'core-dump'.

The system in question uses dnsmasq as local DNS server. Logging that is probably unrelated, but often seen (possibly due to the setup with dnsmasq) is:
dec 21 10:22:00 marshall systemd-resolved[3183]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
dec 21 10:17:15 marshall systemd-resolved[3183]: Using degraded feature set (TCP) for DNS server 127.0.0.1.

# dpkg --status systemd
Package: systemd
Installed-Size: 12444
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: foreign
Version: 237-3ubuntu10.9

Tags: bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
tags: added: bionic
Revision history for this message
Sebastien Bacher (seb128) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

Changed in systemd (Ubuntu):
importance: Undecided → High
Revision history for this message
OlivierS (olivier-olivier) wrote :
Download full text (6.0 KiB)

not sure if this gives helpful information, but here is a backtrace:

# gdb --pid=26353
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 26353
Reading symbols from target:/lib/systemd/systemd-resolved...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu//libc-2.27.so...done.
done.
Reading symbols from target:/lib/systemd/libsystemd-shared-237.so...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu//libm-2.27.so...done.
done.
Reading symbols from target:/lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu//librt-2.27.so...done.
done.
Reading symbols from target:/lib/x86_64-linux-gnu/libcap.so.2...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libacl.so.1...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libcryptsetup.so.12...(no debugging symbols found)...done.
Reading symbols from target:/usr/lib/x86_64-linux-gnu/libip4tc.so.0...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libseccomp.so.2...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
Reading symbols from target:/usr/lib/x86_64-linux-gnu/liblz4.so.1...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libblkid.so.1...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/28/c6aade70b2d40d1f0f3d0a1a0cad1ab816448f.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from target:/lib/x86_64-linux-gnu/libattr.s...

Read more...

Revision history for this message
OlivierS (olivier-olivier) wrote :

the backtrace is not always the same, here another one:

Program received signal SIGSEGV, Segmentation fault.
0x00007fbc408726a6 in sd_event_source_unref () from target:/lib/systemd/libsystemd-shared-237.so
(gdb) bt
#0 0x00007fbc408726a6 in sd_event_source_unref () from target:/lib/systemd/libsystemd-shared-237.so
#1 0x0000556a5158ea02 in ?? ()
#2 0x0000556a5158ea80 in ?? ()
#3 0x0000556a5158ec0b in ?? ()
#4 0x00007fbc4079d41e in ?? () from target:/lib/systemd/libsystemd-shared-237.so
#5 0x00007fbc4079d69a in sd_event_dispatch () from target:/lib/systemd/libsystemd-shared-237.so
#6 0x00007fbc4079d829 in sd_event_run () from target:/lib/systemd/libsystemd-shared-237.so
#7 0x00007fbc4079da6b in sd_event_loop () from target:/lib/systemd/libsystemd-shared-237.so
#8 0x0000556a5158972b in ?? ()
#9 0x00007fbc40b99b97 in __libc_start_main (main=0x556a51588300, argc=1, argv=0x7ffd22f87e28, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7ffd22f87e18) at ../csu/libc-start.c:310
#10 0x0000556a51589dfa in ?? ()

Revision history for this message
Tim Riker (timriker) wrote :

How should I persistently bypass systemd-resolved ? The DNS failures are causes other applications to fail. I'd like resolv.conf to update with dhcp returned data for DNS servers, but, for the time being, I'd like applications to talk to the upstream DNS servers until this issue is resolved.

Revision history for this message
Danny Brugman (dbrugman72) wrote :

I seem to be affected by the same issue. I get regular DNS query time-outs coinciding with a lot of the following log messages (syslog):

systemd-resolved[1119]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

Eventually systemd-resolved crashes. This happens more frequently (/or only) when connected to wifi.

Trace extracted from an Apport crash report (cause was a segmentation fault):

#0 0x00007efeaa3c62ed in sd_event_source_unref (p=0x1b01030036100326) at ../src/libsystemd/sd-event/sd-event.c:1912
        __PRETTY_FUNCTION__ = "sd_event_source_unref"
#1 0x0000561894e7f162 in dns_query_stop (q=q@entry=0x5618967255b0) at ../src/resolve/resolved-dns-query.c:336
        c = <optimized out>
        __PRETTY_FUNCTION__ = "dns_query_stop"
#2 0x0000561894e7f1d0 in dns_query_complete (q=0x5618967255b0, state=<optimized out>) at ../src/resolve/resolved-dns-query.c:528
        __PRETTY_FUNCTION__ = "dns_query_complete"
#3 0x0000561894e7f33f in on_query_timeout (s=<optimized out>, usec=<optimized out>, userdata=<optimized out>) at ../src/resolve/resolved-dns-query.c:539
        q = <optimized out>
        __PRETTY_FUNCTION__ = "on_query_timeout"
#4 0x00007efeaa3ceb76 in source_dispatch (s=s@entry=0x561896722680) at ../src/libsystemd/sd-event/sd-event.c:3201
        saved_type = SOURCE_TIME_BOOTTIME
        r = <optimized out>
        __PRETTY_FUNCTION__ = "source_dispatch"
        __func__ = "source_dispatch"
#5 0x00007efeaa3ceef1 in sd_event_dispatch (e=e@entry=0x5618966cfcd0) at ../src/libsystemd/sd-event/sd-event.c:3634
        ref = <optimized out>
        p = <optimized out>
        r = <optimized out>
        __PRETTY_FUNCTION__ = "sd_event_dispatch"
#6 0x00007efeaa3d0928 in sd_event_run (e=e@entry=0x5618966cfcd0, timeout=timeout@entry=18446744073709551615) at ../src/libsystemd/sd-event/sd-event.c:3692
        r = 1
        __PRETTY_FUNCTION__ = "sd_event_run"
#7 0x00007efeaa3d0b4f in sd_event_loop (e=0x5618966cfcd0) at ../src/libsystemd/sd-event/sd-event.c:3714
        ref = 0x5618966cfcd0
        r = <optimized out>
        __PRETTY_FUNCTION__ = "sd_event_loop"
#8 0x0000561894e5a22a in run (argv=<optimized out>, argc=<optimized out>) at ../src/resolve/resolved.c:84
        notify_stop = 0x561894ea3d70 "STOPPING=1\nSTATUS=Shutting down..."

Log statement from the kernel log:

traps: systemd-resolve[1162] general protection fault ip:7efeaa3c62ed sp:7ffef7d6f1b0 error:0 in libsystemd-shared-245.so[7efeaa39e000+174000]

(Although the trace started with a notify_stop, the system was not shutdown at the time)

Other details:

Release: Ubuntu 20.04.01 (amd64)
Systemd package version: 245.4-4ubuntu3.4

Revision history for this message
Dan Streetman (ddstreet) wrote :

please reopen if this is still an issue

Changed in systemd (Ubuntu):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.