kernel: WARNING: at net/core/dev.c:2402 skb_warn_bad_offload

[c4rl]

Hi,

I have a small server acting as a router/gateway (KVM) and some machines (physical and virtual) are using it to access the internet, however, every time a physycal machine tries to acess the internet this error appears in the /var/log/messages and the connection hangs for this client. I tried to change some options in the module configuration without sucess.

Gateway
kernel: 3.10.0-514.10.2.el7.x86_64
CentOS: 7.3.1611 (Core)

Host where the gateway is:
kernel: 3.10.0-327.18.2.el7.x86_64
CentOS: 7.2.1511 (Core)

Code: Select all

Mar 22 14:14:55 mygw kernel: WARNING: at net/core/dev.c:2402 skb_warn_bad_offload+0xcd/0xda()
Mar 22 14:14:55 mygw kernel: virtio_net: caps=(0x00000001001b0a29, 0x0000000000000000) len=1434 data_len=1306 gso_size=1368 gso_type=5 ip_summed=1
Mar 22 14:14:55 mygw kernel: Modules linked in: nfsv3 nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd virtio_balloon sg parport_pc pcspkr parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
Mar 22 14:14:55 mygw kernel: xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk cirrus crct10dif_pclmul crct10dif_common drm_kms_helper syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw i2c_core floppy virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
Mar 22 14:14:55 mygw kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W      ------------   3.10.0-514.10.2.el7.x86_64 #1
Mar 22 14:14:55 mygw kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Mar 22 14:14:55 mygw kernel: ffff8801460839a8 fe19640e85887e24 ffff880146083960 ffffffff816864ef
Mar 22 14:14:55 mygw kernel: ffff880146083998 ffffffff81085940 ffff88013d2a2f00 ffff88013bb16000
Mar 22 14:14:55 mygw kernel: 0000000000000005 0000000000000001 0000000000000000 ffff880146083a00
Mar 22 14:14:55 mygw kernel: Call Trace:
Mar 22 14:14:55 mygw kernel: <IRQ>  [<ffffffff816864ef>] dump_stack+0x19/0x1b
Mar 22 14:14:55 mygw kernel: [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
Mar 22 14:14:55 mygw kernel: [<ffffffff810859dc>] warn_slowpath_fmt+0x5c/0x80
Mar 22 14:14:55 mygw kernel: [<ffffffff8131e833>] ? ___ratelimit+0x93/0x100
Mar 22 14:14:55 mygw kernel: [<ffffffff81688f2a>] skb_warn_bad_offload+0xcd/0xda
Mar 22 14:14:55 mygw kernel: [<ffffffff81571609>] __skb_gso_segment+0x79/0xb0
Mar 22 14:14:55 mygw kernel: [<ffffffff815719d5>] validate_xmit_skb.part.94+0x135/0x2f0
Mar 22 14:14:55 mygw kernel: [<ffffffff8157217d>] __dev_queue_xmit+0x4cd/0x570
Mar 22 14:14:55 mygw kernel: [<ffffffff81572230>] dev_queue_xmit+0x10/0x20
Mar 22 14:14:55 mygw kernel: [<ffffffff815b5c66>] ip_finish_output+0x466/0x750
Mar 22 14:14:55 mygw kernel: [<ffffffff815b6c63>] ip_output+0x73/0xe0
Mar 22 14:14:55 mygw kernel: [<ffffffff815b5800>] ? ip_fragment.constprop.54+0x90/0x90
Mar 22 14:14:55 mygw kernel: [<ffffffff815b2716>] ip_forward_finish+0x66/0x80
Mar 22 14:14:55 mygw kernel: [<ffffffff815b2a97>] ip_forward+0x367/0x470
Mar 22 14:14:55 mygw kernel: [<ffffffff815b26b0>] ? ip_frag_mem+0x40/0x40
Mar 22 14:14:55 mygw kernel: [<ffffffff815b06fa>] ip_rcv_finish+0x8a/0x350
Mar 22 14:14:55 mygw kernel: [<ffffffff815b1086>] ip_rcv+0x2b6/0x410
Mar 22 14:14:55 mygw kernel: [<ffffffff815b0670>] ? inet_del_offload+0x40/0x40
Mar 22 14:14:55 mygw kernel: [<ffffffff8156fab2>] __netif_receive_skb_core+0x582/0x800
Mar 22 14:14:55 mygw kernel: [<ffffffff815dc074>] ? tcp4_gro_receive+0x134/0x1b0
Mar 22 14:14:55 mygw kernel: [<ffffffff8156fd48>] __netif_receive_skb+0x18/0x60
Mar 22 14:14:55 mygw kernel: [<ffffffff8156fdd0>] netif_receive_skb_internal+0x40/0xc0
Mar 22 14:14:55 mygw kernel: [<ffffffff81570f58>] napi_gro_receive+0xd8/0x130
Mar 22 14:14:55 mygw kernel: [<ffffffffa01c59d5>] virtnet_poll+0x265/0x750 [virtio_net]
Mar 22 14:14:55 mygw kernel: [<ffffffff815705e0>] net_rx_action+0x170/0x380
Mar 22 14:14:55 mygw kernel: [<ffffffff8108f2cf>] __do_softirq+0xef/0x280
Mar 22 14:14:55 mygw kernel: [<ffffffff8169859c>] call_softirq+0x1c/0x30
Mar 22 14:14:55 mygw kernel: [<ffffffff8102d365>] do_softirq+0x65/0xa0
Mar 22 14:14:55 mygw kernel: [<ffffffff8108f665>] irq_exit+0x115/0x120
Mar 22 14:14:55 mygw kernel: [<ffffffff81699138>] do_IRQ+0x58/0xf0
Mar 22 14:14:55 mygw kernel: [<ffffffff8168e2ad>] common_interrupt+0x6d/0x6d
Mar 22 14:14:55 mygw kernel: <EOI>  [<ffffffff81060fe6>] ? native_safe_halt+0x6/0x10
Mar 22 14:14:55 mygw kernel: [<ffffffff810347bf>] default_idle+0x1f/0xc0
Mar 22 14:14:55 mygw kernel: [<ffffffff81035106>] arch_cpu_idle+0x26/0x30
Mar 22 14:14:55 mygw kernel: [<ffffffff810e7e65>] cpu_startup_entry+0x245/0x290
Mar 22 14:14:55 mygw kernel: [<ffffffff8104f07a>] start_secondary+0x1ba/0x230
Mar 22 14:14:55 mygw kernel: ---[ end trace 964a44e7232c7bca ]---

[TrevorH]

Try using ethtool to turn off some of the offload options - use ethtool -k $interface to see them and what is currently enabled and what is not. Use ethtool -K to turn them off (or on).

[c4rl]

After many attempts disabling options with ethtool, once the LRO - Large receive offload - was disabled then the network started to work nicely. It's important to know that parameter must be changed in the *host* interfaces.

Code: Select all

# ethtool -K eth0 lro off

This is the type of bug that only appears if you use a combination of things, they are: 2 NICs -> team or bond -> vlan -> bridge -> NAT, using KVM.


Thanks!