CentOS

# stap -v -e 'probe syscall.read {printf("syscall %s arguments %s \n", name, argstr); exit()}'
Pass 1: parsed user script and 111 library script(s) using 206992virt/34660res/3252shr/31744data kb, in 140usr/10sys/149real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/syscalls2.stp24
source: probe __syscall.read = kernel.function("sys_read").call
^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/2.6.32-754.el6.x86_64/build'

semantic error: while resolving probe point: identifier '__syscall' at 2
source: __syscall.read
^

semantic error: no match

semantic error: while resolving probe point: identifier 'syscall' at <input>:1:
source: probe syscall.read {printf("syscall %s arguments %s \n", name, argstr); exit()}
^

semantic error: no match

... checking online it seems that the solution for this is os debug symbols, but I have no idea how to get them installed. I've tried installing kernel-devel, kernel-debuginfo, kernel-debug ... what am I missing as re. systemtap?

'/lib/modules/2.6.32-754.el6.x86_64/build'

That appears to be the original 6.10 kernel and there have been numerous updates since that. If you are running one kernel and have a newer one installed then they will not match and things like systemtap will not work. The current 6.10 kernel is 2.6.32-754.24.3.el6 and all of your kernel* packages need to be that exact version and also, uname -r needs to report that exact version too.

$ uname -r
2.6.32-754.el6.x86_64


Should I update something? What ... and how?

Hmm...


$ rpm -qa | grep kern | grep devel
kernel-devel-2.6.32-754.24.3.el6.x86_64
kernel-devel-2.6.32-754.el6.x86_64
kernel-devel-2.6.32-754.23.1.el6.x86_64


This shows three different versions of the kernel-devel package, but ALL match the uname result:
$ uname -r
2.6.32-754.el6.x86_64

Which doesn't include anything after the -754 ... so how do I find out if the .24.3 or the .23.1 (or neither) is appropriate?

there is also a stap-prep script that is supposed to supply some of these things, but it has a problem where it wants to read from, I guess, an installation cd?


file:///media/CentOS/repodata/repomd.xml: [Errno 14] Could not open/read file:///media/CentOS/repodata/repomd.xml
Trying other mirror.
file:///media/cdrecorder/repodata/repomd.xml: [Errno 14] Could not open/read file:///media/cdrecorder/repodata/repomd.xml
Trying other mirror.
file:///media/cdrom/repodata/repomd.xml: [Errno 14] Could not open/read file:///media/cdrom/repodata/repomd.xml
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: c6-media. Please verify its path and try again

This shows three different versions of the kernel-devel package, but ALL match the uname result:

Three different versions of the kernel but only one of them matches. 2.6.32-754.el6.x86_64 is kernel-devel-2.6.32-754.el6.x86_64, the other 2 are later kernels that you are not running. You need to reboot into the latest one of those so that you are running a kernel that matches the latest -devel you have installed. Reboot into kernel-2.6.32-754.24.3.el6.x86_64 and run uname -r to check that it is the correct one. You almost certainly want to use debuginfo-install kernel-2.6.32-754.24.3.el6.x86_64 to install the debuginfo required for that kernel. Though I just tried that and it fails as that version of the debuginfo isn't there yet (I just bugged the person responsible so hopefully it'll be there by the time you look). The older debuginfo-install kernel-2.6.32-754.23.1.el6.x86_64 does work.

I don't know if/how I can reboot into one of the others. Since most things tend to match the shorter number (which is what uname -r returns) I think anything with a different number was installed by mistake. I removed the ones with the longer, more detailed numbers.

How would I check what versions I have available to boot into?

what happens if you have multiples of a package installed?
I have access to a different machine where uname -r returns
2.6.32-754.23.1.el6.x86_64

and rpm -qa | grep kernel-dev
kernel-devel-2.6.32-754.11.1.el6.x86_64
kernel-devel-2.6.32-754.17.1.el6.x86_64
kernel-devel-2.6.32-696.el6.x86_64
kernel-devel-2.6.32-754.15.3.el6.x86_64
kernel-devel-2.6.32-754.23.1.el6.x86_64

So ... I have limited authority to add or remove packages on this machine. I would have to clear it with someone else. But ... if the 23.1 package is installed (it is) will it automatically use it if we are booted into the 23.1 kernel (current situation) ? Or do the other versions of the package just kind of confuse things?

I removed the ones with the longer, more detailed numbers.

Wrong way round. The newer ones have longer, higher numbers and contain security fixes that you are now missing. The latest kernel should always be the default selected one when you reboot. If you aim to remove kernels then use package-cleanup --oldkernels --count 1 to remove all but the latest 1 (or 2 or whatever number you give).