no destination available LVS problem

Issues related to applications and software problems
Post Reply
Mslga
Posts: 4
Joined: 2015/07/27 06:27:06

no destination available LVS problem

Post by Mslga » 2015/07/27 06:47:17

Good afternoon!

I had a problem on the server.

The server is configured as a load balancer to proxy sguid.

I used sowfware:
centos 6
2.6.32-504.16.2.el6.x86_64
piranha-0.8.6-4.el6_5.2.x86_64
cat lvs.cf
service = lvs
primary = 10.157.200.15
backup_active = 0

virtual proxylb {
address = 10.157.200.9 eth2:1
vip_nmask = 255.255.255.0
active = 1
load_monitor = none
timeout = 5
reentry = 10
port = 3128

server proxygbstb {
address = 10.157.200.25
active = 1
weight = 1
}

server proxygbprod {
address = 10.157.200.27
active = 1
weight = 1
}
}
on squid server i used iptables
-A PREROUTING -d 10.157.200.9/32 -p tcp -m tcp --dport 3128 -j REDIRECT
\

today i'd a trouble with my squid servers
they are very slow process users requests

in the LVS logs, I saw these:
Jul 27 09:14:35 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:35 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:35 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:36 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:36 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:36 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:36 proxylb kernel: IPVS: WLC: no destination available
Jul 27 09:14:36 proxylb kernel: IPVS: WLC: no destination available
ipvsadm -L is showing too many active and inactive connections

I reboot the server and his work has recovered

that could cause problems?

p.s.
I apologize for any errors in the letter, but my English is bad

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: no destination available LVS problem

Post by aks » 2015/07/27 18:18:40

ipvsadm -L is showing too many active and inactive connections
How many? Are you running out of available ports?

Mslga
Posts: 4
Joined: 2015/07/27 06:27:06

Re: no destination available LVS problem

Post by Mslga » 2015/07/28 06:26:27

Hi,

now it show correct quantity of connections
[root@proxylb ha]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP proxygb:squid wlc
-> 10.157.200.25:squid Route 1 408 2129
-> 10.157.200.27:squid Route 1 411 1894
How many? Are you running out of available ports?
Yesterday, ipvsadm showed the other values:
[root@proxylb ha]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP proxygb:squid wlc
-> 10.157.200.25:squid Route 1 13589 16258
-> 10.157.200.27:squid Route 1 14213 12001

Yesterday, the ports of squid was available. network problems were not.

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: no destination available LVS problem

Post by aks » 2015/07/28 18:00:24

Okay I don't understand what you meant by your last message.

Please post output from:
netstat -antp | wc -l

Mslga
Posts: 4
Joined: 2015/07/27 06:27:06

Re: no destination available LVS problem

Post by Mslga » 2015/07/29 03:48:24

Okay I don't understand what you meant by your last message.

Please post output from:
netstat -antp | wc -l
sorry
:oops:
[root@proxylb ha]# netstat -antp | wc -l
29

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: no destination available LVS problem

Post by aks » 2015/07/30 17:01:54

Okay so 29 is not much. The question is how many ports where in use before the reboot?
The only clue we have is
proxylb kernel: IPVS: WLC: no destination available
As far as I remember WLC is the scheduler.
I think one of the LVS load balancing nodes was invalid when the error occurred.
Are there any clues in keepalived or the ldirectord logs?

Mslga
Posts: 4
Joined: 2015/07/27 06:27:06

Re: no destination available LVS problem

Post by Mslga » 2015/07/31 03:51:52

29 july logs
Jul 29 08:44:38 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 29 08:44:48 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 29 09:06:53 proxylb nanny[1565]: [inactive] shutting down 10.157.200.25:3128 due to connection failure
Jul 29 09:07:08 proxylb nanny[1565]: [ active ] making 10.157.200.25:3128 available
Jul 29 09:09:43 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 29 09:09:53 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 29 10:56:14 proxylb nanny[1565]: [inactive] shutting down 10.157.200.25:3128 due to connection failure
Jul 29 10:56:24 proxylb nanny[1565]: [ active ] making 10.157.200.25:3128 available
Jul 29 11:04:59 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 29 11:05:04 proxylb nanny[1565]: [inactive] shutting down 10.157.200.25:3128 due to connection failure
Jul 29 11:05:04 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:04 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:04 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:04 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:04 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:05 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:05 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:05 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:05 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:05 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:05:09 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 29 11:05:14 proxylb nanny[1565]: [ active ] making 10.157.200.25:3128 available
Jul 29 11:38:29 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 29 11:38:34 proxylb nanny[1565]: [inactive] shutting down 10.157.200.25:3128 due to connection failure
Jul 29 11:38:35 proxylb kernel: __ratelimit: 67 callbacks suppressed
Jul 29 11:38:35 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:38:35 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:38:35 proxylb kernel: IPVS: WLC: no destination available
Jul 29 11:38:35 proxylb kernel: IPVS: WLC: no destination available
30 july logs
Jul 30 09:17:51 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 11:53:02 proxylb nanny[1566]: [inactive] shutting down 10.157.200:3128 due to connection failure
Jul 30 11:53:12 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 13:54:46 proxylb auditd[1275]: Audit daemon rotating log files
Jul 30 15:23:38 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 15:23:48 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 15:43:43 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 15:43:53 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 15:58:48 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 15:58:58 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 16:53:59 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 16:54:09 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 16:58:59 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 16:59:09 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 17:09:04 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 17:09:14 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 17:54:14 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
Jul 30 17:54:24 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
today's logs
Jul 31 07:16:37 proxylb auditd[1275]: Audit daemon rotating log files
Jul 31 09:44:25 proxylb kernel: device eth2 left promiscuous mode

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: no destination available LVS problem

Post by aks » 2015/07/31 20:11:12

Jul 30 17:09:14 proxylb nanny[1566]: [ active ] making 10.157.200.27:3128 available
Jul 30 17:54:14 proxylb nanny[1566]: [inactive] shutting down 10.157.200.27:3128 due to connection failure
You seem to have intermittent connectivity issues. Anything in the switch/router along along the network path (i.e.: any intermediate network device)? Is the software running on port 3128 on node 10.157.200.27 crashing and restarting?

Post Reply