Experiencing high packet loss on Vmware VM

Issues related to configuring your network
Post Reply
Vard0
Posts: 3
Joined: 2018/07/06 05:33:41

Experiencing high packet loss on Vmware VM

Post by Vard0 » 2018/07/06 06:33:37

Hi.

I'm running a Centos 7 server on a Vmware VM server and it is experiencing high packet loss. (30% to 40%) both inbound and outbound.

Other servers on the same VMWare host are not experiencing the problem so I don't think it is originating from the VMware platform.

I have tried switching the NIC from VMXNET3 to E1000 and that doesn't seem to have made a difference.

I have also tried increasing the buffer size and swiching to Jumbo packets.

ethtool -G ens160 rx 4096 tx 4096
ethtool -G ens192 rx-jumbo 2048

Neither of these seems to have done the trick.

i am pretty sure that this started happening after a Yum update.

I'm not sure what to try next.

CentOS Linux release 7.5.1804 (Core)
NAME="CentOS Linux"

Thanks

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: Experiencing high packet loss on Vmware VM

Post by pjsr2 » 2018/07/06 08:08:02

You are not giving a lot of information on your network setup. My understanding is that you have configured two network interfaces, named ens160 and ens192.

Please provide the output of

Code: Select all

ip a
route -n
You should define GATEWAY only for one network interface. Could it be that you set GATEWAY for multiple interfaces?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Experiencing high packet loss on Vmware VM

Post by TrevorH » 2018/07/06 10:06:54

Good idea. Do your two interfaces have ip addresses within the same subnet as each other?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Vard0
Posts: 3
Joined: 2018/07/06 05:33:41

Re: Experiencing high packet loss on Vmware VM

Post by Vard0 » 2018/07/06 10:20:25

Thanks very much for your reply.

On the Vmware settings I only have one NIC installed :

ens160

Jumbo packets was applied to this NIC as well. (The command I posted was before I edited with the correct NIC name. My apologies).

The results of the commands you requested are below. I'm pretty surpised by the results. It looks like there could remnants of the VMXNET3 NIC.

Is that correct?

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens160: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:50:56:9f:79:56 brd ff:ff:ff:ff:ff:ff
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:9f:79:56 brd ff:ff:ff:ff:ff:ff
inet 10.0.50.222/24 brd 10.0.50.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe9f:7956/64 scope link
valid_lft forever preferred_lft forever

route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.50.250 0.0.0.0 UG 100 0 0 ens33
10.0.50.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Experiencing high packet loss on Vmware VM

Post by TrevorH » 2018/07/06 10:56:12

Your current active interface is ens33 not ens160.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Vard0
Posts: 3
Joined: 2018/07/06 05:33:41

Re: Experiencing high packet loss on Vmware VM

Post by Vard0 » 2018/07/06 11:27:53

Thanks TrevorH

Ok, I have removed the E1000 NIC and we are back to the VMXNET3

The problem is still there:-

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:9f:79:56 brd ff:ff:ff:ff:ff:ff
inet 10.0.50.222/24 brd 10.0.50.255 scope global noprefixroute ens160
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe9f:7956/64 scope link
valid_lft forever preferred_lft forever

route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.50.250 0.0.0.0 UG 100 0 0 ens160
10.0.50.0 0.0.0.0 255.255.255.0 U 100 0 0 ens160

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: Experiencing high packet loss on Vmware VM

Post by pjsr2 » 2018/07/06 13:07:16

Your MAC address is fishy. The first three bytes ( 00:50:56: ) are OK. However, the fourth byte is 9F. It is expected to be between 00 and 3F.

See https://kb.vmware.com/s/article/219

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Experiencing high packet loss on Vmware VM

Post by TrevorH » 2018/07/06 13:10:43

Personally I'd have removed the vmxnet3 interface and left the e1000. Never had any good experiences with vmxnet3.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: Experiencing high packet loss on Vmware VM

Post by pjsr2 » 2018/07/06 15:31:28

Main reasons for high package loss are:
(1) Physically damaged/bad cabling
(2) To much network traffic
(3) Routing conflicts.

(1) and (2) can be excluded if this is the only VM that shows the problems.
Leaves you with (3). So, look for conflicting IP addresses or MAC addresses on your network. A MAC address that starts with 00:50:56:9f is in the range that VMware reserves for internal usage. So your MAC address could potentially collide with some internal MAC address of VMware.

When you had two network cards configured both had the same MAC address:
2: ens160: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:50:56:9f:79:56 brd ff:ff:ff:ff:ff:ff
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:9f:79:56 brd ff:ff:ff:ff:ff:ff
Having two network cards with the same MAC address is definitely wrong. Luckely, since you deleted one that problem is gone.

By default VMware assigns random last octets in the MAC addresses, so I suspect that you have set this MAC address manually. Try setting another MAC address (perhaps let VMware assign a default one) and test if this solves your package drop issue. Unless you have some stupid software license check that is tied to the MAC address, you should be able to change the MAC address without further consequences.

If changing the MAC address does not resolve your issue, carefully check your network for multiple machines with the same IP address.

At this stage, don't worry about the jumbo frame sizes. Better leave it at the defaults so you are only changing one thing at a time. The (automatic) scaling of frame sizes that is normally done by the network layer will be disturbed by the package drop anyhow.

I know that TrevorH is not fond of vmxnet3. Personally, I've never had problems with vmxnet3. Changing the interface type is not what I would attemp at this point.

Post Reply