Glusterfs blues

Issues related to applications and software problems
hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Glusterfs blues

Post by hpux » 2012/02/07 05:45:58

Installed glusterfs from epel had it working fairly well.

Installed some other rpms to make multimedia work better.

Now glusterfs does not want to work.

Feb 6 22:29:25 solaria abrtd: Package 'glusterfs' isn't signed with proper key
Feb 6 22:29:25 solaria abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2012-02-06-22:29:24-4090 (res:2), deleting


Crashes every time I start the glusterd daemon.

Should I have installed from epel or is this now part of the core functionality of the OS?

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Glusterfs blues

Post by pschaff » 2012/02/07 18:59:13

EPEL has a policy of not replacing distro packages, and they are the only source for glusterfs ins of the [url=http://wiki.centos.org/AdditionalResources/Repositories]Repositories[/url] page. What repo did you use for multimedia packages? What does "rpm -V glusterfs" show?

Please [url=http://www.centos.org/modules/newbb/viewtopic.php?topic_id=28723&forum=54]provide more information about your system[/url] by running "./getinfo.sh package" and showing us the output file.

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/08 02:01:26

Information for general problems.
[code]
== BEGIN uname -rmi ==
2.6.32-220.4.1.el6.x86_64 x86_64 x86_64
== END uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
epel-release-6-5.noarch
centos-release-6-2.el6.centos.7.x86_64
adobe-release-x86_64-1.0-1.noarch
rpmforge-release-0.5.2-2.el6.rf.x86_64
== END rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS release 6.2 (Final)
== END cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END getenforce ==

== BEGIN free -m ==
total used free shared buffers cached

used atrpms to do the multi media stuff.

Backed some of it out but not completely.

I've disabled the repo and yum removed and yum installed glusterfs

Still no joy.

ran yum clean all after yum remove.

No joy after re-install.

[root@solaria bin]# rpm -qa | grep gluster
glusterfs-vim-3.2.5-6.el6.x86_64
glusterfs-3.2.5-6.el6.x86_64
glusterfs-server-3.2.5-6.el6.x86_64
glusterfs-rdma-3.2.5-6.el6.x86_64
glusterfs-fuse-3.2.5-6.el6.x86_64
glusterfs-geo-replication-3.2.5-6.el6.x86_64


[root@solaria bin]# uname -a
Linux solaria 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

Feb 7 20:49:13 solaria GlusterFS[4079]: [2012-02-07 20:49:13.141263] C [rdma.c:3934:rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
Feb 7 20:49:16 solaria abrt[4124]: saved core dump of pid 4079 (/usr/sbin/glusterfsd) to /var/spool/abrt/ccpp-2012-02-07-20:49:16-4079.new/coredump (44335104 bytes)
Feb 7 20:49:16 solaria abrtd: Directory 'ccpp-2012-02-07-20:49:16-4079' creation detected
Feb 7 20:49:17 solaria abrtd: Package 'glusterfs' isn't signed with proper key
Feb 7 20:49:17 solaria abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2012-02-07-20:49:16-4079 (res:2), deleting


[root@solaria ccpp-2012-02-07-05:49:21-4586]# ll
total 32832
-rw-r-----. 1 abrt invest 4 Feb 7 05:49 analyzer
-rw-r-----. 1 abrt invest 6 Feb 7 05:49 architecture
-rw-r-----. 1 abrt invest 30 Feb 7 05:49 cmdline
-rw-r-----. 1 abrt invest 7 Feb 7 05:49 component
-rw-r-----. 1 abrt invest 297582592 Feb 7 05:49 coredump
-rw-r-----. 1 abrt invest 1 Feb 7 05:49 count
-rw-r-----. 1 abrt invest 14844 Feb 7 05:49 dso_list
-rw-r-----. 1 abrt invest 2155 Feb 7 05:49 environ
-rw-r-----. 1 abrt invest 30 Feb 7 05:49 executable
-rw-r-----. 1 abrt invest 7 Feb 7 05:49 hostname
-rw-r-----. 1 abrt invest 25 Feb 7 05:49 kernel
-rw-r-----. 1 abrt invest 63386 Feb 7 05:49 maps
-rw-r-----. 1 abrt invest 26 Feb 7 05:49 os_release
-rw-r-----. 1 abrt invest 27 Feb 7 05:49 package
-rw-r-----. 1 abrt invest 71 Feb 7 05:49 reason
-rw-r-----. 1 abrt invest 22 Feb 7 05:49 reported_to
-rw-r-----. 1 abrt invest 1309744 Feb 7 05:49 sosreport.tar.xz
-rw-r-----. 1 abrt invest 10 Feb 7 05:49 time
-rw-r-----. 1 abrt invest 3 Feb 7 05:49 uid
-rw-r-----. 1 abrt invest 7 Feb 7 05:49 username
-rw-r-----. 1 abrt invest 40 Feb 7 05:49 uuid
-rw-r-----. 1 abrt invest 178 Feb 7 05:49 var_log_messages
[root@solaria ccpp-2012-02-07-05:49:21-4586]#
[root@solaria ccpp-2012-02-07-05:49:21-4586]# pwd
/var/spool/abrt/ccpp-2012-02-07-05:49:21-4586

glusterfs -V
glusterfs 3.2.5 built on Jan 10 2012 18:19:36
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc.
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/08 03:14:31

crash dump

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/08 17:44:56

This is a very vanilla 64 bit Centos system.

I have two 6.2 64 bit VM's on the bugger and glusterfs works on them.

I can't believe configuring the system to be able to watch videos (or rather listen) while I work could cause this.

I'm about ready to trash the system and start over. Something else funky could have happened here.

Anyone have a clue on what I can do besides re-install? I can take care of backing up the data and imaging the system in an hour using kickstart but its quite irritating.

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Glusterfs blues

Post by pschaff » 2012/02/08 18:23:54

Reinstalling is usually overkill, but is at least guaranteed to get you back to a deterministic state. You did not show but the beginning of the requested "./getinfo.sh package" output so it is not clear how your repos are configured.

To see what repos you have installed packages from:
[code]
rpm -qa --qf "%{VENDOR} \n" | sort | uniq
[/code]
The result should be a list of VENDOR tags. To see the packages from a particular Vendor, for example EPEL:[code]
rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep "Fedora Project" | cut -d ' ' -f 1 | sort
[/code]
The result should be a list of EPEL packages.

To show RPMforge packages:[code]
rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep "Dag Apt" | cut -d ' ' -f 1 | sort
[/code]
To show ATrpms packages:[code]
rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep ATrpms | cut -d ' ' -f 1 | sort
[/code]

Removing the ATrpms packages, if you suspect that is the problem, may be worth trying. If core packages have been replaced you may instead need to downgrade to the CentOS versions.

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/09 02:24:07

########################
Results are in the file:
* /tmp/basedata.oNXlxf *
########################

To provide this information in support of a question on the CentOS fora,
mailing list, or IRC channel please post its contents to

http://pastebin.centos.org/

or another similar public site of your choice.
Select a retention time longer than the default of 'a day'
and provide a link to the information in your post.

Alternatively, post in-line, trimming to remove any unnecessary information
for your topic. For forum posts, please leave in the
[code] ... [/code] tags to preserve formatting.

WARNING - Check the contents of the /tmp/basedata.oNXlxf file to ensure it does
not contain any sensitive information such as public IP addresses or hostnames.
If necessary obfuscate such information before posting. For example:
146.xxx.xxx.41 or myhost.mydomain.net
[root@solaria bin]# cat /tmp/basedata.oNXlxf
Information for package problems.
[code]
== BEGIN uname -rmi ==
2.6.32-220.4.1.el6.x86_64 x86_64 x86_64
== END uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
epel-release-6-5.noarch
centos-release-6-2.el6.centos.7.x86_64
adobe-release-x86_64-1.0-1.noarch
rpmforge-release-0.5.2-2.el6.rf.x86_64
== END rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS release 6.2 (Final)
== END cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END getenforce ==

== BEGIN free -m ==
total used free shared buffers cached
Mem: 5759 1703 4055 0 36 388
-/+ buffers/cache: 1278 4480
Swap: 16383 0 16383
== END free -m ==

== BEGIN rpm -qa yum\* rpm-\* python | sort ==
python-2.6.6-29.el6.x86_64
rpm-build-4.8.0-19.el6.x86_64
rpm-libs-4.8.0-19.el6.x86_64
rpm-python-4.8.0-19.el6.x86_64
yum-3.2.29-22.el6.centos.noarch
yum-metadata-parser-1.1.2-16.el6.x86_64
yum-plugin-fastestmirror-1.1.30-10.el6.noarch
yum-plugin-security-1.1.30-10.el6.noarch
yum-utils-1.1.30-10.el6.noarch
== END rpm -qa yum\* rpm-\* python | sort ==

== BEGIN ls /etc/yum.repos.d ==
adobe-linux-x86_64.repo
atrpms-bleeding.repo
atrpms.repo
atrpms-testing.repo
CentOS-Base.repo
CentOS-Debuginfo.repo
CentOS-Media.repo
epel.repo
epel-testing.repo
google-chrome.repo
google.repo
mirrors-rpmforge
mirrors-rpmforge-extras
mirrors-rpmforge-testing
rpmforge.repo
== END ls /etc/yum.repos.d ==

== BEGIN cat /etc/yum.conf ==
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=16&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
== END cat /etc/yum.conf ==

== BEGIN yum repolist all ==
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
* base: mirror.nexcess.net
* epel: mirrors.ptd.net
* extras: mirror.steadfast.net
* rpmforge: fr2.rpmfind.net
* updates: pubmirrors.reflected.net
repo id repo name status
adobe-linux-x86_64 Adobe Systems Incorporated enabled: 2
atrpms Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-bleeding Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-bleeding-debuginfo Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-bleeding-source Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-debuginfo Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-source Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-testing Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-testing-debuginfo Red Hat Enterprise Linux 6 - x86_64 - A disabled
atrpms-testing-source Red Hat Enterprise Linux 6 - x86_64 - A disabled
base CentOS-6 - Base enabled: 6,294
c6-media CentOS-6 - Media disabled
centosplus CentOS-6 - Plus disabled
contrib CentOS-6 - Contrib disabled
debug CentOS-6 - Debuginfo disabled
epel Extra Packages for Enterprise Linux 6 - enabled: 6,916
epel-debuginfo Extra Packages for Enterprise Linux 6 - disabled
epel-source Extra Packages for Enterprise Linux 6 - disabled
epel-testing Extra Packages for Enterprise Linux 6 - disabled
epel-testing-debuginfo Extra Packages for Enterprise Linux 6 - disabled
epel-testing-source Extra Packages for Enterprise Linux 6 - disabled
exodus exodus-viewer enabled: 3
extras CentOS-6 - Extras enabled: 4
google-chrome google-chrome - 64-bit enabled: 3
rpmforge RHEL 6 - RPMforge.net - dag enabled: 4,197
rpmforge-extras RHEL 6 - RPMforge.net - extras disabled
rpmforge-testing RHEL 6 - RPMforge.net - testing disabled
updates CentOS-6 - Updates enabled: 338
repolist: 17,757
== END yum repolist all ==

== BEGIN egrep 'include|exclude' /etc/yum.repos.d/*.repo ==
== END egrep 'include|exclude' /etc/yum.repos.d/*.repo ==

== BEGIN sed -n -e "/^\[/h; /priority *=/{ G; s/\n/ /; s/ity=/ity = /; p }" /etc/yum.repos.d/*.repo | sort -k3n ==
== END sed -n -e "/^\[/h; /priority *=/{ G; s/\n/ /; s/ity=/ity = /; p }" /etc/yum.repos.d/*.repo | sort -k3n ==

== BEGIN rpm -qa kernel\* | sort ==
kernel-2.6.32-220.2.1.el6.x86_64
kernel-2.6.32-220.4.1.el6.x86_64
kernel-2.6.32-220.el6.x86_64
kernel-devel-2.6.32-220.2.1.el6.x86_64
kernel-devel-2.6.32-220.4.1.el6.x86_64
kernel-devel-2.6.32-220.el6.x86_64
kernel-firmware-2.6.32-220.4.1.el6.noarch
kernel-headers-2.6.32-220.4.1.el6.x86_64
== END rpm -qa kernel\* | sort ==
[/code][Moderator edit: Restored [i]code[/i] tag to preserve formatting.]

[root@solaria bin]# rpm -qa --qf "%{VENDOR} \n" | sort | uniq
Adobe Systems Inc.
ATrpms.net
CentOS
Dag Apt Repository, http://dag.wieers.com/apt/
Fedora Project
Fujitsu Limited
Google Inc.
(none)
Red Hat
Red Hat, Inc.

[root@solaria bin]# rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep "Fedora Project" | cut -d ' ' -f 1 | sort
dirac-libs
directfb
epel-release
jack-audio-connection-kit
libcdaudio
libkate
libmodplug
mod_security
openvpn
openvpn-auth-ldap
portaudio
transmission
transmission-cli
transmission-common
transmission-gtk
[root@solaria bin]#


[root@solaria bin]# clear;rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep "Dag Apt" | cut -d ' ' -f 1 | sort





a52dec
aalib
amrwb
glib
libdvdplay
libmad
libmpeg2
librtmp
libsidplay
lsdvd
orc
pkcs11-helper
rpmforge-release
schroedinger
twolame


[root@solaria bin]#
[root@solaria bin]# rpm -qa --qf "%{NAME} %{VENDOR} \n" | grep ATrpms | cut -d ' ' -f 1 | sort
atrpms-repo
enca
gecko-mediaplayer
gnome-mplayer
gstreamer-plugins-ugly
gstreamer-vaapi
gtk+
libavcodec52
libavutil50
libcaca
libdca0
libdvdcss
libdvdcss2
libdvdnav
libfaac0
libfaad2
liblirc_client0
libmikmod
libmms
libmp3lame0
libnemesi
libopencore-amrnb0
libopencore-amrwb0
libquicktime0
libva1
libva-glx1
libva-x11_1
libvdpau1
libx264_115
libx264_118
libxavs1
libxosd2
libxvidcore4
mjpegtools
mpg123
mplayer
mplayer-fonts
mplayer-skins
nas-libs
netembryo
openal-soft
svgalib
tslib
xmms-libs

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Glusterfs blues

Post by pschaff » 2012/02/09 14:09:37

We really did not need to see the screen output from the script, and the yum diagnostics were meant for your consumption, but do show quite a dangerous mix of packages. Without careful use of yum-plugin-priorities (which is not installed at all) and/or considerably repo tuning with includepkgs/exclude, that mix is of 3rd party repos is pretty much a guaranteed recipe for disaster.

Frankly, if you want to run a cluster filesystem, and presumably a cluster, I'd leave [b]all[/b] the multimedia stuff out of the mix and use a separate system for that.

A reinstall would be the cleanest path forward, but to attempt fix things at this point, I'd probably try removing all the 3rd party packages, disabling all 3rd party repos, reinstalling anything necessary that was removed along with the 3rd party packages, enabling only EPEL, and reinstalling glusterfs.

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/09 19:21:57

Well I'll get yum-priorities installed ASAP.

Other advice welcome. I re-imaged the box but am wondering if there is a better source for glusterfs than epel. I thought RH built that into the base.

So far glusterfs is stable

hpux
Posts: 25
Joined: 2012/02/07 05:42:30
Contact:

Re: Glusterfs blues

Post by hpux » 2012/02/09 19:43:32

What do you think about rpmforce?

A monitor free box with a near identical setup (and yum-priorities) running in my basement without a monitor (negating the need to entertain the owner) remained stable.



[quote]
hpux wrote:
Well I'll get yum-priorities installed ASAP.

Other advice welcome. I re-imaged the box but am wondering if there is a better source for glusterfs than epel. I thought RH built that into the base.

So far glusterfs is stable[/quote]

Post Reply