REDIS SERVER is not able to start after node reboot

General support questions
z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

REDIS SERVER is not able to start after node reboot

Post by z_haseeb » 2017/12/12 09:13:37

We configured a 2 nodes cluster thru pcs on RHEL 7.4. Cluster is configured and running fine. I tried to make a redis Master/Slave resource which also created fine and running as Master/Slave. However when I restarted Active node, the REDIS instance become Master on Passive node of HA but when the rebooted node become available, the REDIS shows stopped at pcs status. I tried multiple versions of Redis

stevemowbray
Posts: 519
Joined: 2012/06/26 14:20:47

Re: REDIS SERVER is not able to start after node reboot

Post by stevemowbray » 2017/12/12 09:21:37

If you have RHEL you should ask Red Hat, that's why you pay for support. CentOS is not RHEL.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: REDIS SERVER is not able to start after node reboot

Post by TrevorH » 2017/12/12 10:30:51

And that sounds like it's working as designed. You'd need to tell pcs to switch it back when the original node comes back but most people don't do that as it involves an unnecessary and risky switch back to the original node.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Can we make a master/slave resource with systemd:redis agent

Post by z_haseeb » 2017/12/12 14:58:12

Can we make a master/slave resource with systemd:redis agent ? Below is my experience with systemd:redis agent
- I am able to make a resource with this command pcs resource create REDIS systemd:redis successfully but when I further run this command pcs resource master MASTERSLAVE REDIS then my REDIS resource disappeared.

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: REDIS SERVER is not able to start after node reboot

Post by hunter86_bg » 2017/12/12 16:48:51

Can you try to run 'pcs resource debug-start redis' in order to check why it doesn't start in slave mode.
Also, how did you define the master resource?

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Re: REDIS SERVER is not able to start after node reboot

Post by z_haseeb » 2017/12/13 05:43:01

hunter86_bg wrote:Can you try to run 'pcs resource debug-start redis' in order to check why it doesn't start in slave mode.
Also, how did you define the master resource?
pcs resource create REDIS systemd:redis

# pcs status
Cluster name: xxxCLUSTER
Stack: corosync
Current DC: xxxCOM1 (version 1.1.16-12.el7-94ff4df) - partition with quorum
Last updated: Wed Dec 13 10:38:08 2017
Last change: Wed Dec 13 10:38:01 2017 by root via cibadmin on xxxCOM2

2 nodes configured
1 resource configured

Online: [ xxxCOM1 xxxCOM2 ]

Full list of resources:

REDIS (systemd:redis): Started xxxCOM1


# pcs resource master MASTERSLAVE REDIS


# pcs status
Cluster name: xxxCLUSTER
Stack: corosync
Current DC: xxxCOM1 (version 1.1.16-12.el7-94ff4df) - partition with quorum
Last updated: Wed Dec 13 10:38:29 2017
Last change: Wed Dec 13 10:38:26 2017 by root via cibadmin on xxxCOM2

2 nodes configured
0 resources configured

Online: [ xxxCOM1 xxxCOM2 ]

No resources



# pcs resource debug-start REDIS
Must supply a resource id with -r
Error performing operation: No such device or address


################
################

# pcs cluster cib|grep REDIS
<nvpair id="redis_replication-REDIS_REPL_INFO" name="REDIS_REPL_INFO" value="xxxCOM2"/>
<primitive class="systemd" id="REDIS" type="redis">
<op id="REDIS-monitor-interval-60" interval="60" name="monitor" timeout="100"/>
<op id="REDIS-start-interval-0s" interval="0s" name="start" timeout="100"/>
<op id="REDIS-stop-interval-0s" interval="0s" name="stop" timeout="100"/>

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: REDIS SERVER is not able to start after node reboot

Post by hunter86_bg » 2017/12/14 05:03:42

Try adding the I'd with 'PCs resource debug-start redis -r redis'

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Re: REDIS SERVER is not able to start after node reboot

Post by z_haseeb » 2017/12/14 07:42:13

]# pcs resource debug-start redis -r redis pcs resource debug-start REDIS -r redis pcs resource debug-start redis -r REDIS
(ABOVE THREE COMMANDS HAVE SAME RESULT)
option -r not recognized

Usage: pcs [-f file] [-h] [commands]...
Control and configure pacemaker and corosync.

Options:
-h, --help Display usage and exit.
-f file Perform actions on file instead of active CIB.
--debug Print all network traffic and external commands run.
--version Print pcs version information.
--request-timeout Timeout for each outgoing request to another node in
seconds. Default is 60s.

Commands:
cluster Configure cluster options and nodes.
resource Manage cluster resources.
stonith Manage fence devices.
constraint Manage resource constraints.
property Manage pacemaker properties.
acl Manage pacemaker access control lists.
qdevice Manage quorum device provider on the local host.
quorum Manage cluster quorum settings.
booth Manage booth (cluster ticket manager).
status View cluster status.
config View and manage cluster configuration.
pcsd Manage pcs daemon.
node Manage cluster nodes.
alert Manage pacemaker alerts.


[root@xxx5TPE1 ~]# pcs resource debug-start redis
Error: unable to find a resource/clone/master/group/bundle: redis
[root@xxx5TPE1 ~]# pcs resource debug-start REDIS
Operation start for REDIS (systemd::redis) returned 0

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Re: REDIS SERVER is not able to start after node reboot

Post by z_haseeb » 2017/12/18 15:47:15

Should I make a new post ?

I also tried to use ocf:heartbeat:redis agent for redis as Master/Slave. The agent configured successfully. When I rebooted the Master redis node, the standby HA node become Master redis node successfully. However when the rebooted node become available, it shows I am Master node and redis also show stopped . I have found below web post which seems my issue is actually a bug. If the below link is related to my problem then please guide how can I utilize the patch mentioned in below link
https://bugzilla.redhat.com/show_bug.cgi?id=1283240

z_haseeb
Posts: 102
Joined: 2009/12/31 07:58:30

Re: REDIS SERVER is not able to start after node reboot

Post by z_haseeb » 2018/01/01 11:01:45

My problem is resolved today thru a workaround.

I added a delay in redis service under below(Service) section

[Service]
ExecStartPre=/bin/sleep 30

ABOVE SERVICE SECTION IS EXTRACTED FROM THE BELOW SERVICE PATH
systemctl status redis | grep Loaded
Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)

Post Reply