Help creating custom service init script for Apache Kafka

[chrisneal]

Hi everyone.

I'm having an issue creating my init script for Apache Kafka that I'm hoping someone can help me with. I've spent about 6 hours on it now, and I'm at a loss.

The problem I'm having is that the Java program that starts a Kafka server doesn't by default return control to the shell. It runs in the foreground. I couldn't find a way to call the Java class to have it return, so the logical thing to try was to throw an "&" at the end of the 'exec' line. While from a shell script this works as expected, when done within the service framework, it behaves differently. It *appears* as if the call to 'exec' with an "&" at the end somehow bypasses the service control and doesn't let the "start" section actually finish. It's a bit hard to describe. I've attached the init file to this post in hopes that there is something simple I'm doing wrong.

Here is the 'start' section of the script also:

Code: Select all

start() {
    checkJava
    [ -x $exec ] || exit 5
    [ -f $CONF_FILE ] || exit 6
    if [ -n "$MAX_OPEN_FILES" ]; then
        ulimit -n $MAX_OPEN_FILES
    fi
    echo -n $"Starting $prog: "
    # if not running, start it up here, usually something like "daemon $exec"
    daemon $daemonopts $exec daemon kafkaServer kafka.Kafka $CONF_FILE &
    retval=$?
    echo $retval
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

Here is what happens when I run it with the & in the script:

Code: Select all

root@n6[297]:/etc/init.d> service kafka status
kafka is stopped
root@n6[298]:/etc/init.d> service kafka start
Starting kafka: 0
root@n6[299]:/etc/init.d> service kafka status
kafka dead but subsys locked

Here is what happens without the "&":

Code: Select all

root@n6[434]:/etc/init.d> service kafka status
kafka is stopped
root@n6[435]:/etc/init.d> service kafka start
Starting kafka: 

If I do a ps, I can see that it did start, just not return:

Code: Select all

root@n6[425]:~> ps -ef | grep kafka
root      1631  1062  0 21:50 pts/2    00:00:00 /usr/java/bin/java -Xmx512M -Xms512M -server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka/kafka/bin/../logs/zookeeper-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dlog4j.configuration=file:bin/../config/log4j.properties -cp :/opt/kafka/kafka/bin/../core/target/scala-2.8.0/*.jar:/opt/kafka/kafka/bin/../perf/target/scala-2.8.0/kafka*.jar:/opt/kafka/kafka/bin/../libs/jopt-simple-3.2.jar:/opt/kafka/kafka/bin/../libs/log4j-1.2.15.jar:/opt/kafka/kafka/bin/../libs/metrics-annotation-2.2.0.jar:/opt/kafka/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka/bin/../libs/scala-compiler.jar:/opt/kafka/kafka/bin/../libs/scala-library.jar:/opt/kafka/kafka/bin/../libs/slf4j-api-1.7.2.jar:/opt/kafka/kafka/bin/../libs/slf4j-simple-1.6.4.jar:/opt/kafka/kafka/bin/../libs/snappy-java-1.0.4.1.jar:/opt/kafka/kafka/bin/../libs/zkclient-0.3.jar:/opt/kafka/kafka/bin/../libs/zookeeper-3.3.4.jar:/opt/kafka/kafka/bin/../kafka_2.8.0-0.8.0.jar org.apache.zookeeper.server.quorum.QuorumPeerMain config/zookeeper.properties
root      1661  1045  0 21:51 pts/1    00:00:00 /bin/sh /sbin/service kafka start
root      1666  1661  0 21:51 pts/1    00:00:00 /bin/sh /etc/init.d/kafka start
root      1672  1666  0 21:51 pts/1    00:00:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /opt/kafka/kafka/bin/kafka-run-class.sh daemon kafkaServer kafka.Kafka /opt/kafka/kafka/config/server.properties
root      1673  1672  2 21:51 pts/1    00:00:01 java -Xmx256M -server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8888 -Dlog4j.configuration=file:/opt/kafka/kafka/config/log4j.properties -cp :/opt/kafka/kafka/bin/../core/target/scala-2.8.0/*.jar:/opt/kafka/kafka/bin/../perf/target/scala-2.8.0/kafka*.jar:/opt/kafka/kafka/bin/../libs/jopt-simple-3.2.jar:/opt/kafka/kafka/bin/../libs/log4j-1.2.15.jar:/opt/kafka/kafka/bin/../libs/metrics-annotation-2.2.0.jar:/opt/kafka/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka/bin/../libs/scala-compiler.jar:/opt/kafka/kafka/bin/../libs/scala-library.jar:/opt/kafka/kafka/bin/../libs/slf4j-api-1.7.2.jar:/opt/kafka/kafka/bin/../libs/slf4j-simple-1.6.4.jar:/opt/kafka/kafka/bin/../libs/snappy-java-1.0.4.1.jar:/opt/kafka/kafka/bin/../libs/zkclient-0.3.jar:/opt/kafka/kafka/bin/../libs/zookeeper-3.3.4.jar:/opt/kafka/kafka/bin/../kafka_2.8.0-0.8.0.jar kafka.Kafka /opt/kafka/kafka/config/server.properties
root      1722  1708  0 21:51 pts/0    00:00:00 grep kafka

I can also see the service processes still running as well....

Help is much appreciated. I know this can be done.

Thank you for your help.
Chris

[chrisneal]

After more investigation, it appears that with the & added to the script, the daemon function in /etc/init.d/functions never writes the pid file, so "status" reports incorrectly. I suppose that's a work around to force the PID file to get written, but I'd sure like to know what's really going on here.

So, in summary, I added a "<&- &" to the exec line of the kafka-run-class.sh script:

exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" <&- &

Then things started:

Code: Select all

root@n6[500]:/etc/init.d> service kafka status
kafka is stopped
root@n6[501]:/etc/init.d> service kafka start
Starting kafka:                                            [  OK  ]

Then I have to manually trap the PID within the init script, write it to the PID file myself, then status returns properly, and stop works properly.

Code: Select all

root@n6[504]:/etc/init.d> service kafka status
kafka (pid  5217) is running...
root@n6[505]:/etc/init.d> service kafka stop
Stopping kafka:                                            [  OK  ]

Yucky work around.

Still would appreciate input.
Chris

[TrevorH]

Unsure if this is what you're looking for

Code: Select all

# yum info jakarta-commons-daemon-jsvc
Loaded plugins: changelog, downloadonly, priorities, refresh-packagekit, security, versionlock
198 packages excluded due to repository priority protections
Available Packages
Name        : jakarta-commons-daemon-jsvc
Arch        : x86_64
Epoch       : 1
Version     : 1.0.1
Release     : 8.9.el6
Size        : 27 k
Repo        : base
Summary     : Java daemon launcher
URL         : http://jakarta.apache.org/commons/daemon/
License     : ASL 2.0
Description : Java daemon launcher.

It provides an executable called jsvc which is specifically for running java as a service but it may require changes to the java program, not sure.

[chrisneal]

Thank you Trevor!

I'll take a look at that.

Chris