CentOS 5.4 with PerfCtr and PAPI

[EdLesMann]

[quote]
[quote]
*Interesting note: Apprently there are problems with hmaccalc that threw errors similar to what I found on the wiki so I added the --without fips as per instructions at [S2-5] and now everything works. It looks like plenty of work is being done to fix this.
[/quote]
The fixed version of the [i]hmaccalc[/i] package can be obtained from the [i]ELRepo project[/i] -- [url=http://www.centos.toracat.org/ned/CentOS-5/testing/hmaccalc/]here[/url] -- Just manually download the correct version for your system architecture and perform an update ([b]rpm -Uvh hmaccalc-*.rpm[/b]). You will now be able to build with [i]fips[/i] checking enabled.
[/quote]
Thanks for the heads up. Researching on if I want/need fips is still on going, but I at least have a place to start now.

[quote]
[quote]
*note: the space between the # and the i386 is needed else it throws errors during rpmbuild...
[/quote]
Absolutely correct. In you opinion, does that need to be clarified in the wiki article?
[/quote]

Well when I was going through it I just added a #i386. I wasn't sure why it was put there and I thought it was just for the sake of having the config file labled. However, when the build failed I noticed it failed looking for a config file that was called 386. So I went hunting for that file and I couldn't find it either. There was a i386 but no 386. This was the only thing that stuck out to me as being wrong so I corrected it by adding a space (and literally no other change at all) and I got past that particular stage. So it was kinda my fault for thinking it was a comment and not thinking about it being a needed part of the puzzle but I do think it may be nice to have that clarification.

[quote]
[quote]
*Note: I still don't understand all these changes, but if I didn't do them I had build problems and the CentOS wiki said to do them.
[/quote]
You should only need to make those changes if you are building your custom kernel with kABI checking [i]disabled[/i].
[/quote]

From what I have read having kABI is good so I want it enabled and I didn't disable it in the rpmbuild command. I will go back and do another recompile and see if I still get the errors I was getting.

Thanks again for all your help in useful information!

[EdLesMann]

[quote]
toracat wrote:
[quote]
AlanBartlett wrote:

I suspect you are referring to the block of code that begins with the comment "[i]#if a rhel kernel, apply the rhel config options[/i]". If so, this is because you are using the [i]C-Plus[/i] sources . . . Let's call [b]toracat[/b] back to comment on this. ;-)
[/quote]
Yes, the centosplus kernel [i]is[/i] a custom kernel after all. :-)

A little history: when RHEL/CentOS-5 came out, CentOS developer, Johnny Hughes, the official maintainer of the cplus kernel, had to scratch his head -- the customization he made disappears as soon as he built the kernel. Soon he found that the block of code we are now talking about had to be removed. The kernel wiki article was, at that time, written for CentOS-4, so he added the note as part of the update to that wiki page.[/quote]

I was kinda scratching my head wondering what I should do since that step was missing. I just took the off chance that it had already been removed and went on my way. It isn't always the case where it works out so well, but it did this time. :-)

Thanks for the info and help!

[EdLesMann]

Alan,
You were right. I just took a fresh rebuild (been doing this a lot recently :-) ) and went through the process again. This time the only change I made in the kernel-2.6.spec file was the 'buildid .perfctr' and it all worked. So those other changes are not needed.

I am also composing a better step-by-step where I take into account several things that I am now learning about. Such as FIPS, even after just a few minutes of research I can already tell I am going to want it enabled. 8-)

When I understand this a bit better and have a better step-by-step process I will post back just so that others who may find this thread don't have to go through the same hair-pulling I did. :-D

Thanks again for all your help guys! It would have taken me much longer with much more frustration without your help!

[EDIT] I just posted the step-by-step to my [url=http://backedbyapenguin.wordpress.com/2010/03/05/centos-5-4-perfctr-and-papi/]blog[/url] so that I am not constantly making changes here on the forums and cluttering it up.

If someone with more resources then I would like to merge these notes else where in the CentOS wiki documentation please feel free to do and let me know. I would actually prefer a more official location for these notes.

[AlanBartlett]

[quote]
So it was kinda my fault for thinking it was a comment and not thinking about it being a needed part of the puzzle but I do think it may be nice to have that clarification.
[/quote]
Done. :-)

[quote]
[EDIT] I just posted the step-by-step to my [url=http://backedbyapenguin.wordpress.com/2010/03/05/centos-5-4-perfctr-and-papi/]blog[/url] so that I am not constantly making changes here on the forums and cluttering it up.
[/quote]
A note has been made to read your blog. ;-)

[quote]
If someone with more resources then I would like to merge these notes else where in the CentOS wiki documentation please feel free to do and let me know. I would actually prefer a more official location for these notes.
[/quote]
Hmm. I can say straight away, without reading them, that they could not be added to the [i]CentOS[/i] wiki [url=http://wiki.centos.org/HowTos/Custom_Kernel]custom kernel[/url] article. As you appreciate, that article details the generic steps to be taken by a kernel builder -- steps that may need to be modified according to each builder's requirements.

[b]toracat[/b], your view / comments, please?

[EdLesMann]

[quote]
AlanBartlett wrote:
[quote]
So it was kinda my fault for thinking it was a comment and not thinking about it being a needed part of the puzzle but I do think it may be nice to have that clarification.
[/quote]
Done. :-)
[/quote]
Thanks!

[quote]
Hmm. I can say straight away, without reading them, that they could not be added to the [i]CentOS[/i] wiki [url=http://wiki.centos.org/HowTos/Custom_Kernel]custom kernel[/url] article. As you appreciate, that article details the generic steps to be taken by a kernel builder -- steps that may need to be modified according to each builder's requirements.
[/quote]

I fully understand. I was just trying to think of a way of getting the info out there in case others run into the same problem. I have got a lot to learn, but I will do what I can to keep the entry updated and working.


Thanks again!

[blade_queen]

Hi, EdLesMann,

Thanks for your instructions. I have followed your instruction and compiled the same centos kernel (the same version except mine is x86_64) with perfctr. The kernel is working however, perfctr doesn't seem working. When I try to run perfex -i, it reports,
perfex: open perfctrs: No such device

When I check the perfctr module at serviceconf, it says,
perfctr is not present in this kernel

I am afraid I didn't patch the perfctr correctly. Here is the output of mine perfctr patching test

~/PerfCtr/perfctr-2.6.40/update-kernel --test --patch=2.6.18-164.el5-redhat

Applying kernel source patch for 2.6.18-164.el5-redhat (forced, kernel is 2.6.18-164.11.1.el5.centos.plus.perfctr1xen)

patching file CREDITS
patching file Documentation/ioctl-number.txt
patching file MAINTAINERS
patching file arch/i386/Kconfig
patching file arch/i386/kernel/entry.S
patching file arch/i386/kernel/i8259.c
patching file arch/i386/kernel/process.c
patching file arch/powerpc/Kconfig
patching file arch/powerpc/kernel/process.c
patching file arch/x86_64/Kconfig
patching file arch/x86_64/kernel/entry.S
patching file arch/x86_64/kernel/i8259.c
patching file arch/x86_64/kernel/process.c
patching file drivers/Makefile
patching file fs/exec.c
patching file include/asm-i386/mach-default/irq_vectors.h
patching file include/asm-i386/mach-visws/irq_vectors.h
patching file include/asm-i386/processor.h
patching file include/asm-i386/system.h
patching file include/asm-powerpc/processor.h
patching file include/asm-x86_64/hw_irq.h
patching file include/asm-x86_64/irq.h
patching file include/asm-x86_64/processor.h
patching file include/asm-x86_64/system.h
patching file include/linux/config.h
patching file kernel/exit.c
patching file kernel/sched.c
patching file kernel/timer.c

Did I miss something? I would appreicate if you would take a look and share your experience. Thanks.

[EdLesMann]

[quote]
blade_queen wrote:
I have followed your instruction and compiled the same centos kernel (the same version except mine is x86_64) with perfctr. The kernel is working however, perfctr doesn't seem working. When I try to run perfex -i, it reports,
perfex: open perfctrs: No such device

When I check the perfctr module at serviceconf, it says,
perfctr is not present in this kernel
[/quote]

Hey sorry about that, I somehow missed your post.

Just a quick update. There are two things wrong with my current instructions which I am currently working on.

1) You have to install perfctr after you are in your new kernel (which you seem to have done if you have perfex)
2) Perfctr doesn't work right for some reason.

The first part is easy enough. The second part is the doozy. I still have not figured it out. This forum has been my only help so far.

My posts to both the guys at PAPI and the Perfctr email went completely unanswered. I can't get _anyone_ to help me build this thing properly from either one of those groups.
I was able to find a old tutorial for fedora 9 or 10 that I was able to get working, but all of the ones I have found for CentOS have either been incomplete or don't work.
I pinged one guy from the Rocks cluster list who built a custom kernel for CentOS 4 and I got no response from him either. The rest of the list seems to say "sorry, but you are on your own".

So where am I at now in all this mess?
Well I am switching my users over to Oprofile. It seems to do a good chunk of what most of them need. At least so far from what I have seen Oprofile has got a million times better support and documentation.
There are a few of my users that seem to really need the perfctr and PAPI, so I installed the old kernel I had from before the update. It didn't really want to install and it complained a lot but a good beatdown with a hammer and 'rpm --force' seemed to do the trick. :lol: I would REALLY like to use a newer kernel so I have not given up, I just have not worked on it in a few days.

The problem seems to be this grouping of messages in the build-err.log file (if you piped out the messages as I did then you will probably find them as well).
[code].config:227:warning: trying to assign nonexistent symbol PERFCTR
.config:228:warning: trying to assign nonexistent symbol KPERFCTR
.config:229:warning: trying to assign nonexistent symbol PERFCTR_DEBUG
.config:230:warning: trying to assign nonexistent symbol PERFCTR_INIT_TESTS
.config:231:warning: trying to assign nonexistent symbol PERFCTR_VIRTUAL
.config:232:warning: trying to assign nonexistent symbol PERFCTR_GLOBAL
.config:233:warning: trying to assign nonexistent symbol PERFCTR_INTERRUPT_SUPPORT
.config:234:warning: trying to assign nonexistent symbol PERFCTR_CPUS_FORBIDDEN_MASK
[/code]

I don't know why they are not getting patched properly. When the 'make oldconfig' is run, it picks up the changes AND adds them to the .config file for you so it isn't like they are erronious entries.

I am sorry I can't help you much right now. There are a bunch of other postings out there going back for several years of others having these same types of problems (being unable to compile perfctr/papi code and get it working) and yet the perfctr/papi guys seem to be silent and not very willing to help. I am stumbling through as best as I can, but before a few weeks ago I had never even attempted something like this before ( I still have a LOT to learn and a long ways to go). I am going through just about every warning and error thrown in the build process with the latest kernel (even found a few things that are completely unreleated to my problem :-D ) and yet still no luck.

Let me dig out my notes about perfctr and I will update that blog posting I posted earlier in the thread. I have been trying to keep that blog posting updated hoping it might help someone else, but there are a few notes I need to adjust. I will start on that now.

Good luck!

[blade_queen]

Hi, EdLesMann,
Thanks a lot for your response. I was a little late with posting too. It's very unfortunate that little support is available from PAPI and perfctr. For my project, oprofile is not sufficient so I have to go this long route. One thing I noticed that the configuration files, .config and configs/*.config in BUILD has been rewritten every time after rpmbuild is called in your last step. Once the configure files in SOURCES are preserved. As a result, the configuration files used for building do not have any PERFCTR information included. Likely, the final kernel doesn't include perfctr. I have little knowledge about rpmbuild. Did you observe the same thing from your build? It is indeed weird.

Joy

[EdLesMann]

Sorry for the late response; I got super busy this week.

[quote]
blade_queen wrote:
One thing I noticed that the configuration files, .config and configs/*.config in BUILD has been rewritten every time after rpmbuild is called in your last step. Once the configure files in SOURCES are preserved. As a result, the configuration files used for building do not have any PERFCTR information included. Likely, the final kernel doesn't include perfctr. I have little knowledge about rpmbuild. Did you observe the same thing from your build?[/quote]

I just took a look at my old build configuration and sure enough, it all gets wiped! None of the changes stick around! :-x

I think I may have a solution, but I may need a couple of days. I will post back as soon as I can.

Thanks for the pointer!

[EdLesMann]

[quote]
[quote]
blade_queen wrote:
One thing I noticed that the configuration files, .config and configs/*.config in BUILD has been rewritten every time after rpmbuild is called in your last step. Once the configure files in SOURCES are preserved. As a result, the configuration files used for building do not have any PERFCTR information included. Likely, the final kernel doesn't include perfctr. I have little knowledge about rpmbuild. Did you observe the same thing from your build?[/quote]

I just took a look at my old build configuration and sure enough, it all gets wiped! None of the changes stick around! :-x
[/quote]

I have just blown 8 hrs today working on this. I don't know whose fault this is but I can create perfctr and papi builds for Fedora and Debian without problems but I can't seem to get CentOS to do it. This is hardcore pissing me off. I don't think I have ever run into so many problems recompiling a kernel _EVER_! It has been over a month now since my first post with countless hours wasted and I still don't have it. Ughh...

Anyway on to the point of this post, I am not 100% on this but here is what I have found.....

If you follow the directions on my blog (slightly updated since the post above, but same idea) you WILL get an RPM of a kernel and it will NOT have perfctr included. Why? I don't really know. If you followed my instructions (which I copied from the CentOS tutorial) you should have a build-err.log. Take a look inside of it.
[code]$ grep PERFCTR build-err.log
.config:227:warning: trying to assign nonexistent symbol PERFCTR
.config:228:warning: trying to assign nonexistent symbol KPERFCTR
.config:229:warning: trying to assign nonexistent symbol PERFCTR_DEBUG
.config:230:warning: trying to assign nonexistent symbol PERFCTR_INIT_TESTS
.config:231:warning: trying to assign nonexistent symbol PERFCTR_VIRTUAL
.config:232:warning: trying to assign nonexistent symbol PERFCTR_GLOBAL
.config:233:warning: trying to assign nonexistent symbol PERFCTR_INTERRUPT_SUPPORT
.config:234:warning: trying to assign nonexistent symbol PERFCTR_CPUS_FORBIDDEN_MASK[/code]

Hrm...well that is probably why the final RPM doesn't have it enabled! OK so what is causing it?

Let me be clear, I don't really have a clue what I am doing. This is all very new to me. Everytime I have had to rebuild a kernel it has always been as easy as extracting, applying a patch, and rebuilding. This has been the first time where things didn't work like that. So I went through the log files as best as i could trying to figure out what in /dev/null was going on. I even started snapshoting the directory structure so I could compare files.

I discovered that on line 8812 of the kernel-2.6.spec file:
[code]make ARCH=$Arch nonint_oldconfig[/code]
This seems to be causing the problems. If you leave this in, then while the compile is going you can look at the files that were patched AND THE PATCH IS NO LONGER THERE! By commenting this line out, the patch stays however the build will fail with errors when it gets to building the modules. Here is the first one.
[code]install: cannot stat `System.map': No such file or directory[/code]
I tried to fix each error one by one (Googling the error and using the results as my fixes) but after who-knows-how-many failures I was too pissed off to continue so I ditched that build...

I started fresh (rm -rf ~/rpmbuild) and I went back to my instructions. I followed them all the way to step 15, where I also put in an extra line that forced the update-kernel rerun on the build after it wiped out the changes but the rpmbuild always wigged out and I could never get it to run successfully. I could be putting it in the wrong place, but again I don't have a clue what I am doing and I am not going to "trial and error" a file that is over 15000 lines long.

So I started fresh again, but this time I took a snapshot of the directory after i ran the `rpmbuild -bp` command. Then I ran the perfctr update-kernel and took another snapshot. Then I did a "diff -urp --unidirectional-new-file orig modified > perfctr.patch" just like is in the Fedora insturctions I found posted online. Then I added the patch to the kernel-2.6.spec file as patch 99998 (there are two places you have to modify). This seemed to work at first as the 'rpmbuild -bp' completes successfully WITH the perfctr patch already in place! Horray! Right? No..
When you run the 'rpmbuild -bb' command it errors out in the build with a generic "make[1]: *** [nonint_oldconfig] Error 1"
Thanks generic error! That totally helps me track down the problem!

Well I vi'd the kernel-2.6.spec file and looked for nonint_oldconfig and found three instances. One is a comment about it being useful for automated builds. The second is the EXACT same line I started the day out messing with to stop deleting my patch! The third appears to be similar to the second.

Well now I am back to exactly the same problem I started out with today! Something about that blasted make command is jacking around with the perfctr patch.

If anyone has any ideas or suggestions I would love to hear them. Again, of ALL the forums/irc channels/ect this forum is the only place I have found anyone willing to even talk to me (even the perfctr guy and the papi guys didn't respond to emails asking for
help). I think that if I can find ONE person who can explain why the patch gets deleted when rpmbuild runs then we can come up with a solution but so far no one has responded to my pleas.

So I am left with yet another day completely wasted and nothing to show for it and therefore I am calling it quits for a while.