Page 1 of 4

CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/03 01:18:12
by EdLesMann
Hey guys!

So I inherited a 'older' system and I was asked to update it. My predecessor left notes that don't quite make sense.
I was able to yum update to 5.4 with various degrees of success and after lots of Googling the box is currently up and running (Horray!). The only thing I have left to do is mod the kernel for PerfCTR and PAPI for the programmers/developers.

Here is where the notes really fall through. The notes seemed like complete gibberish to me at first (and seriously only a few steps with little info on each) but after LOTS of reading I have been able to flush them out a bit. So now I have a process...but it fails. I would greatly appreciate some help figuring out where I am going wrong.

Here are the steps:
1) Download CentOS Plus repo kernel.
Done. I pulled this one: kernel-2.6.18-164.11.1.el5.centos.plus.src.rpm

2) Download PerfCTR and extract.
Done. I found perfctr 2.6.40 here: http://user.it.uu.se/~mikpe/linux/perfctr/2.6/ and I extracted it into my home directory.

3) Install rpm.
His directions said to do that as root, but the CentOS wiki has a big thing on recompiling kernels saying that is a no-no. So as my user (please correct me if i am wrong) I did:
mkdir -p ~/rpmbuild/{BUILD,RPMS,SRPMS,SPECS,SOURCES}
echo '%_topdir /home/USER/rpmbuild' > ~/.rpmmacros
rpm -ivh kernel-2.6.18-164.11.1.el5.centos.plus.src.rpm

4) Append the following to each SOURCES/*.config file
CONFIG_PERFCTR=y
CONFIG_KPERFCTR=y
CONFIG_PERFCTR_GLOBAL=y
CONFIG_PERFCTR_VIRTUAL=y
CONFIG_PERFCTR_INIT_TESTS=y
CONFIG_PERFCTR_INTERRUPT_SUPPORT=y
CONFIG_PERFCTR_CPUS_FORBIDDEN_MASK=y

When I do an `ls *.config` in the SOURCES directory it finds:
kernel-2.6.18-i586.config
kernel-2.6.18-i686.config
kernel-2.6.18-i686-PAE.config
kernel-2.6.18-i686-xen.config
kernel-2.6.18-x86_64.config
kernel-2.6.18-x86_64-xen.config

That seems like an awful lot especially since I am only after the x86_64 on this machine (and only after the i686 on my test machine...figured a test machine would be good since I am learning this :lol: ) but OK. Done. I added those lines to the end of all the files.

5) cd to SPECS directory and edit kernel-2.6.spec file:
cd ~/rpmbuild/SPECS
vi kernel-2.6.spec

It has me change the line from "% define buildid" to "%define buildid .perfctr"
Then it has me find the line "END OF PATCH APPLICATIONS" and add below it (modified for the location/kernel I am using):
/home/USER/PerfCtr/perfctr-2.6.40/update-kernel --patch=2.6.18-164.el5-redhat

Then save and quit.

6) Compile and install the generated rpm. Since this is my test box target is i686. Ultimately I want x86_64.
rpmbuild -bb --with baseonly --without debuginfo --target=i686 kernel-2.6.spec

I run rpmbuild and I get:

Installing new kernel source files
+ cp /home/USER/rpmbuild/SOURCES/COPYING.modules Documentation/
+ mkdir configs
+ for cfg in 'kernel-2.6.18-i?86*.config'
+ cp -f /home/USER/rpmbuild/SOURCES/kernel-2.6.18-i586.config /home/USER/rpmbuild/SOURCES/kernel-2.6.18-i686-PAE.config /home/USER/rpmbuild/SOURCES/kernel-2.6.18-i686-xen.config /home/USER/rpmbuild/SOURCES/kernel-2.6.18-i686.config .
+ rm -f kernel-2.6.18-i686-kdump.config kernel-2.6.18-x86_64-kdump.config
+ rm -f kernel-2.6.18-i586.config
+ rm -f 'kernel-2.6.18-*-debug.config'
+ for i in '*.config'
+ mv kernel-2.6.18-i686-PAE.config .config
++ head -1 .config
++ cut -b 3-
+ Arch=i386
+ make ARCH=i386 nonint_oldconfig
CONFIG_PERFCTR_DEBUG
make[1]: *** [nonint_oldconfig] Error 1
make: *** [nonint_oldconfig] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.89224 (%prep)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.89224 (%prep)


And I have been researching this error for a few hours and I am completely stumped. None of the work arounds I found seem to work (and a few of them made things worse which made me glad for a test box I could rebuild :-) )

I would greatly appreciate someone with more knowledge/skill/experience in these matters to help me out. Can anyone point me out where I am going wrong? Any tips on getting PerfCTR and PAPI up and working?


Thank you all for your time and help!

PS: related but different subject, I have been asked to research Oprofile too. I have not started yet due to above error taking precedence but if anyone has any tidbits of wisdom they would like to pass along to me regarding Oprofile I would gladly listen. :-)

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/03 15:05:06
by AlanBartlett
Welcome to the [i]CentOS[/i] fora.

On reading through your exploits, I can tell you that you have done the correct thing with regards to not performing your package builds as [i]root[/i].

I would like you to please read through the two relevant [i]CentOS[/i] wiki articles, [url=http://wiki.centos.org/HowTos/I_need_the_Kernel_Source]I Need the Kernel Source[/url] and [url=http://wiki.centos.org/HowTos/Custom_Kernel]I Need to Build a Custom Kernel[/url].

You may find it beneficial to perform a trial kernel build [i]before[/i] attempting the customisation process.

My initial hints for your customisation:

Ignore your predecessors instructions --

[quote]
4) Append the following to each SOURCES/*.config file
CONFIG_PERFCTR=y
CONFIG_KPERFCTR=y
CONFIG_PERFCTR_GLOBAL=y
CONFIG_PERFCTR_VIRTUAL=y
CONFIG_PERFCTR_INIT_TESTS=y
CONFIG_PERFCTR_INTERRUPT_SUPPORT=y
CONFIG_PERFCTR_CPUS_FORBIDDEN_MASK=y
[/quote]
-- and when you reach [i]Section 2[/i] of [url=http://wiki.centos.org/HowTos/Custom_Kernel]I Need to Build a Custom Kernel[/url], follow it to --

[quote]
First run [i]make oldconfig[/i]. Now you should run either [i]make menuconfig, make gconfig[/i] or [i]make xconfig[/i] to customise the kernel configuration.
[/quote]
It is at this point that you would configure the seven options, above. Once done, continue following the [i]CentOS[/i] wiki article [i]to the letter[/i].

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/04 04:47:06
by EdLesMann
Thanks so much for responding. I am not there yet, but I am still working on a few options. I just didn't want to let this day pass without saying thanks and letting you know I am still working on it.

I did a lot of reading today and I spent a lot of time watching the compile process happen :-)

I took your advice and tried recompiling the kernel without making any changes. That worked (I didn't really test the final result, but I ended up with an rpm!). So I tried again making my changes. That failed with an error very similar to what I had before.
So I went back to the wiki links you posted and I think I know where I am screwing up. On step 4 of your Custom Kernel link, there is a bunch of stuff it wants me to edit out that doesn't exist in the file I have from this centos plus src kernel. I know you said follow it to the letter, but I kinda skipped that part as it didn't quite line up:-?. At the end of 4 it says to add the patch. Well I don't really have a patch per se. I just have this perfctr update-kernel command that my instructions say to put into the kernel-2.6.spec. I am now thinking that is the failing point for this build.

So I did a bunch more research.
I am pretty sure that my predecessor cut and pasted his instructions from [url=http://74.125.155.132/search?q=cache:EAtuz0boj1IJ:cs.famaf.unc.edu.ar/~nicolasw/Docencia/HPCMMM/rocks-4.2.1-perfctr.html+perfctr+%2B%22update-kernel%22+centos&cd=8&hl=en&ct=clnk&gl=us&client=iceweasel-a]here[/url] (Google Cache). Because that looks almost identical to what he gave me.

So I dug around more and found [url=http://debianclusters.cs.uni.edu/index.php/Adding_Performance_Counters_to_the_Kernel]this[/url] but it is for Debian. So I went back to the Perfctr install instructions which I originally thought were way to vague. Now that I have been digging around this code for a few days I am beginning to understand what each step is for but I am still not there. However, I was able to match the perfctr instructions with what I read for that Debian install and thankfully perfctr has patches that match the centos plus repo kernel! Horray! Right? Not really...It threw a TON of errors saying that the kernel was already patched and asked if I wanted to override. I said yes and went on with the recompile. It failed. So I wiped the whole rpmbuild directory and tried again but this time I said no thinking maybe there were a few files it would patch and it would leave the rest alone. That compile took way longer, but it failed too. Then I got to thinking. If it is saying it is already patched, then maybe I just have to enable the patch! So I tried again but this time I added those 7 lines, made the identification change in the spec file, and jumped right to the rpmbuild....and thats another hour of compile time that ended with failure.

So here is where I am at.
*I still don't know a good method of generating that patch file that the wiki says I am supposed to have.
*I have been using the "2> build-err.log | tee build-out.log" that the instructions have but outside of the final error I don't think there is much there worth looking at. If you really want I can post it to pastebin or something.
*I also suspect that I am going to have to figure out if I want/need kabichk. The Custom Kernel page makes it sound like i want it, but it also sounds like it could be problematic. I have not seen an error that mentioned it, but maybe I will try removing that tomorrow and seeing what happens.
*I have a whole new respect for kernel devs and anyone who can wrap their heads around all this stuff. :-D

I would greatly welcome feedback because I have a feeling that I am beginning to just go in circles...

Also, for my previous statement about oprofile I found [url=http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-oprofile.html]this[/url] which I was able to follow and do A LOT more with when I needed a break from perfctr earlier today. It seems that CentOS has a lot of good documentation which I am really grateful for but I guess very few people care enough about perfctr to write the same level of documentation. I think I just might....if I can ever get it to compile! :lol:

Thanks!

[Edit] I don't know why but I decided to look at that last error one more time (the build where I didn't run or include perfctrs update-kernel command; just added the 7 lines to the config file). It IS a different error, but I don't know if it helps or hurts...
$ tail -5 build-err.log
+ '[' i386 '!=' s390 -o '' '!=' kdump ']'
+ make -s ARCH=i386 INSTALL_MOD_PATH=/var/tmp/kernel-2.6.18-164.11.1.el5.centos.plus.perfctr-root modules_install KERNELRELEASE=2.6.18-164.11.1.el5.centos.plus.perfctr
make: *** No rule to make target `modules_install'. Stop.
error: Bad exit status from /var/tmp/rpm-tmp.1280 (%build)
Bad exit status from /var/tmp/rpm-tmp.1280 (%build)

I am tired and calling it a night, but it looks like I have another point to research tomorrow...

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/04 17:59:04
by AlanBartlett
Hi [b]Ed[/b],

Thank you for updating this thread. It seems, with the links you have provided, that I have a spot of reading to do. :lol: As it is very close to the end of my day here (UTC-land) [i][b]and[/i][/b] I have noticed that you are using the [i]CentOS-Plus[/i] kernel sources, I shall give my co-maintainer of those kernel articles, [b]toracat[/b], a ping.

My reason for doing so is not delegation (ducks to avoid the paw swipe, with claws showing) but because [b]toracat[/b] looks after the [i]C-Plus[/i] kernel and knows of the modifications therein made -- I am only familiar with the standard kernel.

Just in case I've missed it, have you already posted a link to the tarball containing the code you wish to add? If not, please do so.

CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/04 18:18:09
by toracat
Those CentOS kernel wiki articles are valid for both distro kernels and centosplus kernels.

As [b]Alan[/b] pointed out earlier, you should not be manually editing the config files in SOURCES. It is important that running [i]make oldconfig[/i] [b]does not prompt you to answer [/b] any query. In other words, when you type that command, you should get the command line prompt back (after seeing some lines scrolling by). If this does not happen, it would lead to the error you reported in your original post.

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/05 02:52:00
by EdLesMann
SUCCESS!!
And not just some fluke either, I have done two successful builds and have put together my step-by-step which I hope I can get someone else to confirm...

Thanks to both of you for your help and comments! It did actually help!

I would appreciate any feedback anyone has to make this better, but I was able to take a freshly installed CentOS 5.4 box and run these directions the second time through and I ended up with a fresh set of rpms! I have not had the chance to do extensive testing yet, but I will get those who are going to be using this system to do that. 8-) Anyway, here are the steps I did.

References:
S1: I need the Kernel Source http://wiki.centos.org/HowTos/I_need_the_Kernel_Source
S2: I need to Build a Custom Kernel http://wiki.centos.org/HowTos/Custom_Kernel
S3: Perfctr install text (which I didn't find online, but it is in the tar.gz file here: http://user.it.uu.se/~mikpe/linux/perfctr/2.6/perfctr-2.6.40.tar.gz )

01 Install a lot of build tools [S1-1 and S2-1]
yum groupinstall "Development Tools"
yum install hmaccalc
yum install ncurses-devel
yum install kernel-devel
yum install rpm-build redhat-rpm-config unifdef

02 Download PerfCTR, extract to home directory, and check for supported kernels[S3]. I used perfctr-2.6.40.
mkdir ~/PerfCtr
cd ~/PerfCtr
tar -zxvf ~/perfctr-2.6.40.tar.gz
ls ~/PerfCtr/perfctr-2.6.40/patches

03 Download the latest source RPM that matches one of the perfctr kernel patches for the CentOS plus modified kernel.
The output of the ls command in step 2 has "patch-kernel-2.6.18-164.el5-redhat" in it which matches one of the CentOS plus kernels "kernel-2.6.18-164.11.1.el5.centos.plus.src.rpm". So I used that kernel.

04 Configure an RPM build environment [S1-2.1]
echo '%_topdir /home/USER/rpmbuild' > ~/.rpmmacros
mkdir -p ~/rpmbuild/{BUILD,RPMS,SRPMS,SPECS,SOURCES}
05 Install source RPM
rpm -i kernel-2.6.18-164.11.1.el5.centos.plus.src.rpm
06 Change to the SPECS directory and build sources [S1-2.1]
cd ~/rpmbuild/SPECS
rpmbuild -bp --target=`uname -m` --without fips kernel-2.6.spec 2> prep-err.log | tee prep-out.log
*Interesting note: Apprently there are problems with hmaccalc that threw errors similar to what I found on the wiki so I added the --without fips as per instructions at [S2-5] and now everything works. It looks like plenty of work is being done to fix this.
07 Change to the kernel build directory and backup config file [prepping for S2-2]
cd ~/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.i686/
cp configs/kernel-2.6.18-i686.config ~/.
08 Clean the configs files up [S3-1]
make mrproper
09 Test patch [S3-2] and if successful (no errors) then run patch
~/PerfCtr/perfctr-2.6.40/update-kernel --test --patch=2.6.18-164.el5-redhat
~/PerfCtr/perfctr-2.6.40/update-kernel --patch=2.6.18-164.el5-redhat
10 Copy config files:
cp configs/kernel-2.6.18-i686.config .config
11 Setup old config and answer y[S2-2] (I know toracat said this shouldn't ask me any questions, but it did and the guide for perfctr on Fedora 8 said to answer y to all the questions; answering n or just hitting enter caused rpmbuild errors while y on each works)
make oldconfig
12 Add to the .config file as the first line whatever is printed out by `uname -i` (may already exist) [S2-2]. Example:
# i386
*note: the space between the # and the i386 is needed else it throws errors during rpmbuild...
13 Append the following to the .config file
CONFIG_PERFCTR=y
CONFIG_KPERFCTR=y
CONFIG_PERFCTR_GLOBAL=y
CONFIG_PERFCTR_VIRTUAL=y
CONFIG_PERFCTR_INIT_TESTS=y
CONFIG_PERFCTR_INTERRUPT_SUPPORT=y
CONFIG_PERFCTR_CPUS_FORBIDDEN_MASK=y
14 Copy config back and copy to SOURCES [S2-2]:
cp .config configs/kernel-2.6.18-i686.config
cp configs/* ~/rpmbuild/SOURCES
15 Edit spec file [S2-4]
cd ~/rpmbuild/SPECS
vi kernel-2.6.spec
Change line 74: % define buildid
To: %define buildid .perfctr
Showing three lines only need to change blank line on 9094 to match
: %endif
:
: # And save the headers/makefiles etc for building modules against
To: %endif
: touch %{_tmppath}/kernel-$KernelVer-kabideps
: # And save the headers/makefiles etc for building modules against
Showing two lines which will become three lines on 9114
: cp Module.markers $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
: mv $RPM_BUILD_ROOT/kabi_whitelist $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
To: cp Module.markers $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
: %if %{with_kabichk}
: mv $RPM_BUILD_ROOT/kabi_whitelist $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
Showing two lines which will become three lines on 9120
: cp symsets-$KernelVer.tar.gz $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
: # then drop all but the needed Makefiles/Kconfig files
To: cp symsets-$KernelVer.tar.gz $RPM_BUILD_ROOT/lib/modules/$KernelVer/build
: %endif
: # then drop all but the needed Makefiles/Kconfig files
*Note: I still don't understand all these changes, but if I didn't do them I had build problems and the CentOS wiki said to do them. So I did the changes and it worked. So though I don't understand all these changes, they are apparently needed so it is a good thing they are in the instructions. :lol: Also, the CentOS wiki has a list of lines to be commented out that I couldn't find in this kernel src.rpm...that maybe worth checking out from someone who knows what to look for.
16 Compile RPM
rpmbuild -bb --without fips --target=`uname -m` kernel-2.6.spec 2> build-err.log | tee build-out.log


and like I said earlier, everything has worked twice now. I need to do a bit more testing, but I figure that since I have done it a few times the other changes can't be that difficult...*knock on wood*... :-D

Thanks again for your help on this. If there is anything that sticks out about these steps, please comment. I am certainly open to suggestions.

Thanks!

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/05 08:25:43
by toracat
Congrats on you successful rebuild.

I probably did not use enough words in my earlier note but what I meant was that the config file should be prepared in such a way that 'make oldconfig' does not prompt you to enter anything. After you have answered all the questions it asked and saved it, if you ran 'make oldconfig' once again, it would have returned the command line prompt this time. In other words, that newly saved config was "ready to go".

Long ago, CentOS developer Johnny Hughes told me he would run 'make oldconfig' twice. I asked why and he said, "to make sure your config is correct". :-)

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/05 13:39:30
by AlanBartlett
Congratulations from me, too, [b]Ed[/b].

By your concerted hard work, you have saved me a lot of reading -- and experimentation. Thank you. :-D

In my last post, I asked if you had posted the link . . . [b][i]D'oh![/i][/b] Of course you had. :oops:

My comments on selected parts of your procedure --

[quote]
*Interesting note: Apprently there are problems with hmaccalc that threw errors similar to what I found on the wiki so I added the --without fips as per instructions at [S2-5] and now everything works. It looks like plenty of work is being done to fix this.
[/quote]
The fixed version of the [i]hmaccalc[/i] package can be obtained from the [i]ELRepo project[/i] -- [url=http://www.centos.toracat.org/ned/CentOS-5/testing/hmaccalc/]here[/url] -- Just manually download the correct version for your system architecture and perform an update ([b]rpm -Uvh hmaccalc-*.rpm[/b]). You will now be able to build with [i]fips[/i] checking enabled.

[quote]
*note: the space between the # and the i386 is needed else it throws errors during rpmbuild...
[/quote]
Absolutely correct. In you opinion, does that need to be clarified in the wiki article?

[quote]
*Note: I still don't understand all these changes, but if I didn't do them I had build problems and the CentOS wiki said to do them.
[/quote]
You should only need to make those changes if you are building your custom kernel with kABI checking [i]disabled[/i].

[quote]
Also, the CentOS wiki has a list of lines to be commented out that I couldn't find in this kernel src.rpm...that maybe worth checking out from someone who knows what to look for.
[/quote]
I suspect you are referring to the block of code that begins with the comment "[i]#if a rhel kernel, apply the rhel config options[/i]". If so, this is because you are using the [i]C-Plus[/i] sources . . . Let's call [b]toracat[/b] back to comment on this. ;-)

[quote]
16 Compile RPM
rpmbuild -bb --without fips --target=`uname -m` kernel-2.6.spec 2> build-err.log | tee build-out.log
[/quote]
You can save yourself some time if you check the usage of the [i]--with baseonly --without debug --without debuginfo[/i] options. Assuming you do not want the debug packages and only want the base, not xen or (if 32-bit architecture) PAE kernels, of course.

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/05 15:29:56
by toracat
[quote]
AlanBartlett wrote:

I suspect you are referring to the block of code that begins with the comment "[i]#if a rhel kernel, apply the rhel config options[/i]". If so, this is because you are using the [i]C-Plus[/i] sources . . . Let's call [b]toracat[/b] back to comment on this. ;-)
[/quote]
Yes, the centosplus kernel [i]is[/i] a custom kernel after all. :-)

A little history: when RHEL/CentOS-5 came out, CentOS developer, Johnny Hughes, the official maintainer of the cplus kernel, had to scratch his head -- the customization he made disappears as soon as he built the kernel. Soon he found that the block of code we are now talking about had to be removed. The kernel wiki article was, at that time, written for CentOS-4, so he added the note as part of the update to that wiki page.

Re: CentOS 5.4 with PerfCtr and PAPI

Posted: 2010/03/05 18:29:45
by toracat
If you are [i]really[/i] interested in how Johnny got to this fix, the history is written in this [url=http://lists.centos.org/pipermail/centos/2007-April/037549.html]mailing list thread[/url] . :-)