Systemtap should require kernel debug information

Thank you for your bug report. Could you please upload the log files and/or tell me exactly what happened when you tried to run this package. Thanks.

I installed systemtap, and tried the sample from the tutorial: http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/doc/tutorial/strace-open.stp?rev=1.2&content-type=text/plain&cvsroot=systemtap

# stap strace-open.stp
semantic error: libdwfl failure (dwfl_linux_kernel_report_offline): No such file or directory while resolving probe point kernel.function("sys_open")?
semantic error: cannot find kernel debuginfo while resolving probe point kernel.function("sys32_open")?
semantic error: no match for probe point while resolving probe point syscall.open
Pass 2: analysis failed. Try again with more '-v' (verbose) options.

Then with the help of strace, I saw that the stap process wanted to access /boot/vmlinux-2.6.20-15-generic

So I installed the kernel debug package and made the symlink:

# apt-get install linux-image-debug-2.6.20-15-generic
# cd /boot
# ln -s vmlinux-dbg-2.6.20-15-generic vmlinux-2.6.20-15-generic

And afterwards, the strace-open.stp sample works as expected.

Of course, systemtap still can run if you don't have the kernel debug package installed, but you won't be able to use the probes referring to symbol names.

This is on powerpc - to get systemtap working I followed the instruction from the previous post, but I still get errors:

 ./keyhack.stp -v
Pass 1: parsed user script and 52 library script(s) in 1010usr/20sys/1120real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found
WARNING: cannot find module kernel debuginfo: No DWARF information found
semantic error: no match for probe point while resolving probe point kernel.function("kbd_event")
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) in 380usr/150sys/592real ms.
Pass 2: analysis failed. Try again with more '-v' (verbose) options.

% ./top.stp -vv
SystemTap translator/driver (version 0.5.12 built 2007-02-02)
(Using Red Hat elfutils 0.123 libraries.)
Copyright (C) 2005-2006 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
Created temporary directory "/tmp/stap5oOiKJ"
Searched '/usr/share/systemtap/tapset/2.6.20-16-powerpc/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20-16-powerpc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/*.stp', match count 33
Searched '/usr/share/systemtap/tapset/LKET/2.6.20-16-powerpc/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20-16-powerpc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/*.stp', match count 19
Pass 1: parsed user script and 52 library script(s) in 990usr/30sys/1184real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found
WARNING: cannot find module kernel debuginfo: No DWARF information found
semantic error: no match for probe point while resolving probe point kernel.function("sys_*")
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 1 global(s) in 360usr/190sys/609real ms.
Pass 2: analysis failed. Try again with more '-v' (verbose) options.
Running rm -rf /tmp/stap5oOiKJ

Hi Miek,

It may be possible that my instructions above are out of date. You need the linux-image-debug-* package that matches your running kernel. You can get your kernel version with running "uname -a". As of now, I had to install linux-image-debug-2.6.20-16-generic. Of course, the symlink name has to be adapted.

If that doesn't help, you could try running strace, to find what kernel symbol file name stap tries to access:

strace -f -e trace=file -o/tmp/log stap /tmp/your_test.stp
grep vmlinux /tmp/log

I did all that, I've installed the debug-* package and fixed the symlink - but still no go. I might try to run this on my AMD64 system. But on power-pc it looks like a no-go.

Did you try with strace? What information did you get? You can also attach the interesting information there.

Okay, I was typing this when systemtap decided to issue a illegal opcode and crash my server :(

Anyhow. Looks like I got it working by:

apt-get install linux-source-2.6.20
compiling a kernel in /usr/src
setting /lib/modules/`uname -r`/build -> /usr/src/linux-source-2.6.20

and then running top.stp, but then my system crashed...

Miek, it is possible that this is a systemtap or kernel problem
(in which case we'd appreciate hearing more details about the
crash - see http://sourceware.org/systemtap/wiki/HowToReportBugs).

But it is also possible that your rebuilt kernel does not exactly
match the one you're *running*. This situation is sometimes
hard to catch in software, though we plan to improve on this.

It is also possible that the ubuntu snapshot of systemtap is
too old - many bugs have certainly been fixed in the last six
months.

Jordan, symlinking the debug kernel to the running kernel name doesn't work for me. Am I doing something wrong? See bug 115830

Thanks

I confirm that the proposed fix does work under gutsy x86_64.

# ln -s /boot/vmlinux-debug-2.6.22-14-generic /boot/vmlinux-2.6.22-14-generic

I am unable to resolve module functions with systemtap, like:

probe module("hidp").function("hidp_input_report") {

fails with:

WARNING: cannot find module hidp debuginfo: No DWARF information found
..etc.

I have linux-image-debug installed, but it only contains the _kernel_ with debug info, not the _modules_, it seems. Am I off base here, or are the modules with DWARF information simply .. not available in ubuntu?

Jordan,

This is still an issue in Hardy, are you still the correct person for this to be assigned to?

Can the systemtap package effectively do what is described in the http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu page?

$ sudo apt-get install linux-image-debug-generic
$ sudo ln -s /boot/vmlinux-debug-$(uname -r) /lib/modules/$(uname -r)/vmlinux
$ sudo apt-get install linux-headers-generic gcc libcap-dev

After doing that, stap seems to be functioning correctly on hardy now.

The package attached depends upon the mentioned required packages - give it a go. I can't promise it won't break anything - but it would be nice to have it tested. This package is for amd64.

I have the same problem as OH mentioned in <a href="https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/106957/comments/11">comment 11</a>.

What packages contains the symbols for the loadable modules?

Just an addition: There seems to be no linux-image-debug-generic package in Intrepid. Will such a package be added later or should we be using something else for Intrepid?

There is one, but it's very well hidden..... (dont ask why, i dont know too).

The packages containing the vmlinux files are here: http://ddebs.ubuntu.com/pool/main/l/linux/

See also bug 289087 - which is about the missing packages, e.g. "linux-image-debug-generic".
This is documented at http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

systemtap 2.1-1~experimental1 in debian now instructs users to see README.Debian every time stap fails. README.Debian then asks them to run the "stap-prep" command that installs all the required headers and debugging symbols. This has been tested on both Debian and Ubuntu.

This seems like it can be closed, now that stap-prep is the documented way to ensure all the debug packages are installed.