Network latency increases as soon as the CPU is slightly used

[Corsi159]

Hello,

I have a latency issue with one of my servers, running on CentOS 6.5 (2.6.32-431.29.2.el6.x86_64).

It is a virtual machine (running on VMware ESX 5.5) with 16GB RAM, 8 CPUs.

If the machine is not experiencing any load, my average ping response time is at about 0.200ms.

But as soon as the machine has the slightlest load (even a load smaller than 1), the ping response time goes up to 8ms.


The problem is that the application that runs on this CentOS server is poorly designed, and uses a huge amount of small packets. Because of that, loading a screen on this application takes 80 seconds instead of 10...

I can recreate the problem on this server at any time using "stress" :

Code: Select all

 stress -c 1

But I cannot reproduce this behaviour on other CentOS servers. So I have the feeling that there is a parameter, somewhere, that has been changed and is causing the issue.

Using tcpdump on the server, I identified that the latency is always following this scenario :

1) server sends packet to client
2) LATENCY (average 8ms)
3) server receives packet from client
4) server immediately (less than 0.01ms of delay) another packet from client

I have the feeling that when one of the CPU is busy, it's not thinking about checking the network queue anymore. Then it remembers, goes there, and read all the packets that were waiting...

Would anybody have an idea about what parameters I would have to check?

Here are some details about the network interface :

Code: Select all

[root@VMWGLIMSP1 sys]# ethtool -i eth 0
ethtool: bad command line argument(s)
For more information run ethtool -h
[root@VMWGLIMSP1 sys]# ethtool -i eth0
driver: vmxnet3
version: 1.1.30.0-k-NAPI
firmware-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@VMWGLIMSP1 sys]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             512

I attached as well to this post the output of the sysctl -a command.

Thanks in advance for your help!