ndo2db is not running ( NAGIOS )

[Ashish Sood]

Hi,

I am working around to find the solution, but still i unable to find the solution of my problem. i am using centos7 and installed nagiosXi in my system but on the nagios home page i can see a alert which saying that ndo2db is not running, i tried to restart it several time but still the problem is persist.

OUTPUT

[root@localhost subsys]# service ndo2db status
ndo2db is not running but subsystem locked

NDO2db configuration file

[root@localhost subsys]# cat /usr/local/nagios/etc/ndo2db.cfg
#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################


lock_file=/usr/local/nagios/var/ndo2db.lock

ndo2db_user=nagios
ndo2db_group=nagios

socket_type=unix

socket_name=/usr/local/nagios/var/ndo.sock

tcp_port=5668


db_servertype=mysql
db_host=localhost
db_port=3306

db_name=nagios
db_prefix=nagios_

db_user=ndoutils
db_pass=n@gweb



## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time. Most admins will want to trim these tables and keep only a
# certain amount of data in them. The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted. Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.

# Keep timed events for 24 hours
max_timedevents_age=1440

# Keep system commands for 1 week
max_systemcommands_age=10080

# Keep service checks for 1 week
max_servicechecks_age=10080

# Keep host checks for 1 week
max_hostchecks_age=10080

# Keep event handlers for 31 days
max_eventhandlers_age=44640





# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file. OR values together to log multiple
# types of information.
# Values: -1 = Everything
# 0 = Nothing
# 1 = Process info
# 2 = SQL queries

debug_level=0



# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
# 1 = More detailed
# 2 = Very detailed

debug_verbosity=1



# DEBUG FILE
# This option determines where the daemon should write debugging information.

debug_file=/usr/local/nagios/var/ndo2db.debug



# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file. If
# the file grows larger than this size, it will be renamed with a .old
# extension. If a file already exists with a .old extension it will
# automatically be deleted. This helps ensure your disk space usage doesn't
# get out of control when debugging.

max_debug_file_size=1000000

Let me know what other information i can provide you to help you to know the root cause of the problem.

Thanks In Advance
Ashish

[aks]

Did you look in /usr/local/nagios/var/ndo2db.debug?
Perhaps the clues lie there?

[Ashish Sood]

@AKS

I tried to find the file but nothing is exist in the system, also tried to find at other location but no help.

[root@localhost ~]# find / -name ndo2db.debug

[aks]

Okay, so

ndo2db is not running but subsystem locked

suggests that ndo2db started and then exiting in an ungraceful manner. It could be something like it couldn't write to a socket or something. Further along in your posted configuration file we see:

# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file. OR values together to log multiple
# types of information.
# Values: -1 = Everything
# 0 = Nothing
# 1 = Process info
# 2 = SQL queries
debug_level=0

Perhaps not having debug switched on would explain why the debug file does not exist?
I'd suggest switching debug on and looking in the debug file (as listed in your posted configuration). Failing that, I'd try and start the daemon manually and see what's output.
Also it could be a SELinux thing, are you running in permissive mode?

[Ashish Sood]

SELinux is disabled on the system.

Thanks
Ashish

[Ashish Sood]

I have also changed the debug_level=1
after it no debug file( /usr/local/nagios/var/ndo2db.debug) is at the location...

[aks]

First of all, don't disable SELinux - run it in permissive mode until all issues are sorted out, then run it in enforcing mode.
Now on to more questions:
1) Does the directory where the log is supposed to be exist and can the user that's running ndo2db read/write/execute the directory?
2) It appears the nso2db daemon is running in an old sysv type init file (given your querying via service rather than systemctl). Have a look in the script (probably a file called /etc/init.d/ndo2db) to see what the script is trying to execute - and then execute that manually - we many see output to the foreground. If you can't read the script, post it here and I'll have a look for you.

[Ashish Sood]

@Aks
I have set back SELinux to permissive mode

1) I am starting ndo2db from superuser, I don't think i would be any permission issue, although i have given write permission to (/usr/local/nagios) directory.

2)

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: ndo2db
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Nagios NDO2DB Initscript
# Description: Nagios Data Out Daemon
### END INIT INFO

# chkconfig: 345 99 01
#
# File : ndo2db
#
# Author : Jorge Sanchez Aymar (jsanchez@lanchile.cl)
#
# Changelog :
#
# 1999-07-09 Karl DeBisschop <kdebisschop@infoplease.com>
# - setup for autoconf
# - add reload function
# 1999-08-06 Ethan Galstad <egalstad@nagios.org>
# - Added configuration info for use with RedHat's chkconfig tool
# per Fran Boon's suggestion
# 1999-08-13 Jim Popovitch <jimpop@rocketship.com>
# - added variable for nagios/var directory
# - cd into nagios/var directory before creating tmp files on startup
# 1999-08-16 Ethan Galstad <egalstad@nagios.org>
# - Added test for rc.d directory as suggested by Karl DeBisschop
# 2000-07-23 Karl DeBisschop <kdebisschop@users.sourceforge.net>
# - Clean out redhat macros and other dependencies
# 2003-01-11 Ethan Galstad <egalstad@nagios.org>
# - Updated su syntax (Gary Miller)
# 2009-07-11 Hendrik Bäcker <andurin@process-zero.de>
# - Rewrite ndo2db init script, inspired by Sascha Runschke
#
#

status_ndo2db ()
{

pid_ndo2db

if ps -p $Ndo2dbPID > /dev/null 2>&1; then
return 0
else
if test -f $Ndo2dbLockDir/$Ndo2dbLockFile; then
return 2
else
return 1
fi
fi

return 1
}

printstatus_ndo2db()
{
if status_ndo2db $1 $2; then
echo "$servicename (pid $Ndo2dbPID) is running..."
exit 0
elif test $? == 2; then
echo "$servicename is not running but subsystem locked"
exit 1
else
echo "$servicename is not running"
exit 1
fi
}


killproc_ndo2db ()
{

kill $2 $Ndo2dbPID

}


pid_ndo2db ()
{

if test ! -f $Ndo2dbRunFile; then
return 1
fi

Ndo2dbPID=`head -n 1 $Ndo2dbRunFile`
return 0
}


# Source function library
# Solaris doesn't have an rc.d directory, so do a test first
if [ -f /etc/rc.d/init.d/functions ]; then
. /etc/rc.d/init.d/functions
elif [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
fi

servicename=ndo2db
prefix=/usr/local/nagios
exec_prefix=/usr/local/nagios
Ndo2dbBin=/usr/local/nagios/bin/ndo2db
Ndo2dbCfgFile=/usr/local/nagios/etc/ndo2db.cfg
Ndo2dbVarDir=/usr/local/nagios/var
Ndo2dbRunFile=$Ndo2dbVarDir/ndo2db.lock
#Ndo2dbLockDir=/var/lock/subsys
Ndo2dbLockDir=/usr/local/nagiosxi/var/subsys
Ndo2dbLockFile=ndo2db
Ndo2dbUser=nagios
Ndo2dbGroup=nagios


# Check that ndo2db exists.
if [ ! -f $Ndo2dbBin ]; then
echo "Executable file $Ndo2dbBin not found. Exiting."
exit 1
fi

# Check that ndo2db.cfg exists.
if [ ! -f $Ndo2dbCfgFile ]; then
echo "Configuration file $Ndo2dbCfgFile not found. Exiting."
exit 1
fi

# See how we were called.
case "$1" in

start)
status_ndo2db
if [ $? -eq 0 ]; then
echo "$servicename already started..."
exit 1
fi
echo -n "Starting $servicename:"

rm -f $Ndo2dbLockDir/$Ndo2dbLockFile
rm -f $Ndo2dbVarDir/ndo.sock

touch $Ndo2dbRunFile
chown $Ndo2dbUser:$Ndo2dbGroup $Ndo2dbRunFile
$Ndo2dbBin -c $Ndo2dbCfgFile
if [ -d $Ndo2dbLockDir ]; then
touch $Ndo2dbLockDir/$Ndo2dbLockFile;
chown $Ndo2dbUser:$Ndo2dbGroup $Ndo2dbLockDir/$Ndo2dbLockFile;
fi
echo " done."
exit 0
;;

stop)
status_ndo2db
if ! [ $? -eq 0 ]; then
echo "$servicename was not running... could not stop"
exit 1
fi
echo -n "Stopping $servicename: "

pid_ndo2db
killproc_ndo2db ndo2db

# now we have to wait for ndo2db to exit and remove its
# own Ndo2dbRunFile, otherwise a following "start" could
# happen, and then the exiting ndo2db will remove the
# new Ndo2dbRunFile, allowing multiple ndo2db daemons
# to (sooner or later) run - John Sellens
#echo -n 'Waiting for ndo2db to exit .'
for i in 1 2 3 4 5 6 7 8 9 10 ; do
if status_ndo2db > /dev/null; then
echo -n '.'
sleep 1
else
break
fi
done
if status_ndo2db > /dev/null; then
echo ''
echo 'Warning - $servicename did not exit in a timely manner'
else
echo 'done.'
fi

rm -f $Ndo2dbStatusFile $Ndo2dbRunFile $Ndo2dbLockDir/$Ndo2dbLockFile $Ndo2dbCommandFile
;;

status)
printstatus_ndo2db
;;

restart)
$0 stop
$0 start
;;

*)
echo "Usage: $servicename {start|stop|restart|status}"
exit 1
;;

esac

[aks]

So basically the script runs this:
rm -f /var/lock/subsys/ndo2db
rm -f /usr/local/nagios/var/ndo.sock
touch /var/lock/subsys/ndo2db
touch /usr/local/nagios/var/ndo2db.lock
chown nagios:nagios /var/lock/subsys/ndo2db
chown nagios:nagios /usr/local/nagios/var/ndo2db.lock
/usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg # this is the actual launching of the binary.
as the user nagios. Now this depends on how you're setup. As root, can you su - nagios? If you can, do the above and post back the output. If not, you could either run as root (which could mean we need to clean up afterwards) or you could use su. To use su: su - nagios -c "<the command line from above>". So we'd do:
su - nagios -c "rm -f /var/lock/subsys/ndo2db"
su - nagios -c "rm -f /usr/local/nagios/var/ndo.sock"
su - nagios -c "touch /var/lock/subsys/ndo2db"
su - nagios -c "touch /usr/local/nagios/var/ndo2db.lock"
su - nagios -c "chown nagios:nagios /var/lock/subsys/ndo2db"
su - nagios -c "chown nagios:nagios /usr/local/nagios/var/ndo2db.lock"
su - nagios -c "/usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg"

Please post back what you did and ALL output (no matter how insignificant it may seem).

[Ashish Sood]

I did exactly, what you have suggest,

Please find the output below.

[nagios@localhost ~]$ rm -f /usr/local/nagios/var/ndo2db.lock
[nagios@localhost ~]$ touch /var/lock/subsys/ndo2db
touch: cannot touch ‘/var/lock/subsys/ndo2db’: Permission denied
[nagios@localhost ~]$
[nagios@localhost ~]$ su - nagios -c "touch /var/lock/subsys/ndo2db"
Password:
su: Authentication failure
[nagios@localhost ~]$ exit
logout
[root@localhost ~]# su - nagios -c "touch /var/lock/subsys/ndo2db"
touch: cannot touch ‘/var/lock/subsys/ndo2db’: Permission denied
[root@localhost ~]# su - nagios -c "touch /var/lock/subsys/ndo2db"
touch: cannot touch ‘/var/lock/subsys/ndo2db’: Permission denied
[root@localhost ~]# touch /var/lock/subsys/ndo2db"
> ^C
[root@localhost ~]# touch /var/lock/subsys/ndo2db
[root@localhost ~]# touch /usr/local/nagios/var/ndo2db.lock
[root@localhost ~]# chown nagios:nagios /var/lock/subsys/ndo2db
[root@localhost ~]# chown nagios:nagios /usr/local/nagios/var/ndo2db.lock
[root@localhost ~]# /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
Support for the specified database server is either not yet supported, or was not found on your system

Restarting ndo2db service

[root@localhost ~]# service ndo2db restart
Restarting ndo2db (via systemctl): [ OK ]
[root@localhost ~]# service ndo2db status
ndo2db is not running but subsystem locked