Enabling virtual functions on an SR-IOV ethernet adapter

Issues related to configuring your network
sazzle
Posts: 6
Joined: 2019/02/13 12:14:04

Enabling virtual functions on an SR-IOV ethernet adapter

Post by sazzle » 2019/02/13 13:57:18

We have just purchased 2 new Intel I350-T4V2 ethernet adapters to use SR-IOV to enable us to allocate individual virtual functions to virtual machines in our new Centos 7 server. I set up a couple of these cards a few years ago, but I am having real problems getting these to work, no matter what I do I can't get it to enable the virtual functions. The device is recognised by the operating system and I have enabled VT-D and SR-IOV in the BIOS and added 'intel_iommu=on iommu=pt pci=assign-buses' to the boot command line (not sure I need all of those but gradually added them to see if they helped, would like to know exactly what they do?). Rebooted and run modprobe -r igb && modprobe igb max_vfs=8. The biggest clue as to what's wrong is at the next step when I enter this

[root@server1 ~]# echo 8 > /sys/class/net/p5p1/device/sriov_numvfs
-bash: /sys/class/net/p5p1/device/sriov_numvfs: Permission denied

Why is permission denied?? I am the root user. The file sriov_numvfs doesn't actually exist I'm not sure if that matters or not? I feel that the permission denied is because it's not recognising that the card has SR-IOV capabilities or maybe I'm missing a setting to make the OS or the hardware support SR-IOV.

lspci
17:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
17:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
17:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
17:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

uname -a
Linux server1 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Please help I am completely tearing my hair out!!

User avatar
TrevorH
Forum Moderator
Posts: 25590
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by TrevorH » 2019/02/13 16:08:00

The contents of the /proc filesystem is just an interface onto what the kernel knows. If the file sriov_numvfs doesn't exist then you cannot create it manually, it will only exist if the kernel creates it.

Go back through the output of `dmesg` and look to see if you have any errors listed.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

tomkep
Posts: 34
Joined: 2018/04/25 13:30:50

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by tomkep » 2019/02/13 18:01:45

Please check what /sys/class/net/p5p1/device/sriov_totalvfs lists. You cannot create more VFs than the number listed in that file.

For example I have Intel's 82576 based NIC which only allows 7 VFs per PF:

Code: Select all

vesemir:~> cat /sys/class/net/enp2s0f0/device/sriov_totalvfs 
7
But my X553 based LOM can do 63:

Code: Select all

triss:~> cat /sys/class/net/eno1/device/sriov_totalvfs 
63
Now I am a bit surprised by a fact that datasheet (here: https://www.intel.com/content/dam/www/. ... asheet.pdf) lists 8 VFs available for 82576 and I only see 7 of them available in a driver. The same datasheet also lists 8 VFs per I350 but I wouldn't be surprised if they were similarly limited, this is the same driver handling both of these NICs.

What is different on my system is that I am getting different error when creating excessive number of VFs:

Code: Select all

 -bash: echo: write error: Numerical result out of range
but I think it is still worth trying.

If it doesn't help, have a look in dmesg as TrevorH suggested, there may be a clue there.

One last thing - is sysfs by any chance mounted read-only on your system?

sazzle
Posts: 6
Joined: 2019/02/13 12:14:04

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by sazzle » 2019/02/13 19:25:25

Thank you for responding :)

There is no sriov_totalvfs file - bear in mind that I have changed the PCI slot the network card is in so it is now p6p1 instead of p5p1 (thought it was worth a shot but it didn't fix anything)
[root@server1 ~]# cat /sys/class/net/p6p1/device/sriov_totalvfs
cat: /sys/class/net/p6p1/device/sriov_totalvfs: No such file or directory

sysfs is mounted readonly on the system, is this the problem? If so how do I change it?
[root@server1 ~]# grep "[[:space:]]ro[[:space:],]" /proc/mounts
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0

This is the output in dmesg for the 1st PF - I don't see any errors exept for the ACPI errors I have posted below but not sure they are related
[ 2.739515] igb 0000:01:00.0: added PHC on eth2
[ 2.739521] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 2.739525] igb 0000:01:00.0: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:85:5c:44
[ 2.739529] igb 0000:01:00.0: eth2: PBA No: Unknown
[ 2.739532] igb 0000:01:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 2.740066] igb 0000:01:00.1: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface.
[ 2.740759] igb 0000:01:00.1: irq 66 for MSI/MSI-X
[ 2.740844] igb 0000:01:00.1: irq 67 for MSI/MSI-X
[ 2.740967] igb 0000:01:00.1: irq 66 for MSI/MSI-X
[ 2.741032] igb 0000:01:00.1: irq 67 for MSI/MSI-X
[ 2.741069] igb 0000:01:00.1: irq 68 for MSI/MSI-X
[ 2.741098] igb 0000:01:00.1: irq 69 for MSI/MSI-X
[ 2.741126] igb 0000:01:00.1: irq 70 for MSI/MSI-X
[ 2.741155] igb 0000:01:00.1: irq 71 for MSI/MSI-X
[ 2.741184] igb 0000:01:00.1: irq 72 for MSI/MSI-X
[ 2.741215] igb 0000:01:00.1: irq 73 for MSI/MSI-X
[ 2.741249] igb 0000:01:00.1: irq 74 for MSI/MSI-X

[ 5.909557] ACPI Error: No handler for Region [SYSI] (ffff8de034ae6048) [IPMI] (20130517/evregion-162)
[ 5.909564] ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305)
[ 5.909569] ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8de034a9e5a0), AE_NOT_EXIST (20130517/psparse-536)
[ 5.909581] ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8de034a9e500), AE_NOT_EXIST (20130517/psparse-536)
[ 5.909596] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753)

User avatar
TrevorH
Forum Moderator
Posts: 25590
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by TrevorH » 2019/02/13 19:43:17

One of those 8 VFs is probably used for the base adapter itself.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

sazzle
Posts: 6
Joined: 2019/02/13 12:14:04

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by sazzle » 2019/02/13 19:56:26

I was wrong sysfs is not mounted read-only I just entered a grep command I found on google and when I saw the output and saw it contained /sys I immediately thought it was but looking at just cat /proc/mounts shows...
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0

User avatar
TrevorH
Forum Moderator
Posts: 25590
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by TrevorH » 2019/02/13 20:28:24

If the /sys directory is lacking any mention of sriov then I would hazard a guess that it's not enabled or that the driver for that card doesn't support it.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

sazzle
Posts: 6
Joined: 2019/02/13 12:14:04

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by sazzle » 2019/02/13 20:42:35

The card was bought from a normal reseller and not rebranded (I have read that Dell sell the same card and disable SR-IOV) but surely a company just reselling an Intel card wouldn't disable any of its functionality? It is a Dell server but the BIOS has an option to enable SR-IOV, which I have enabled, so I wouldn't have thought the issue was the BIOS.

Are there any tests I can do to find out if it's the BIOS, operating system or card that's not supporting it?

Is igb the driver or is that something different? Might I be able to find another driver for the card that does support it?

Thank you for your help it is most greatly appreciated!!

User avatar
TrevorH
Forum Moderator
Posts: 25590
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by TrevorH » 2019/02/13 21:59:39

There are at least 2 places you need to enable SR-IOV on a Dell machine. There's a per card setting and there's also one that controls whether it's off or on. So you can enable it globally but not on any of the cards and then it won't work or you can enable it in the card but not globally and it still won't work.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

sazzle
Posts: 6
Joined: 2019/02/13 12:14:04

Re: Enabling virtual functions on an SR-IOV ethernet adapter

Post by sazzle » 2019/02/13 22:12:54

I think you have just solved my problem although I am not at work so cannot try it till tomorrow, unfortunately SSH is no good for BIOS settings!!

A quick google search brought up this - https://community.mellanox.com/s/articl ... ort-sr-iov
While not the same server (might be but not sure) or network card, the BIOS screen shots look identical to what I have seen. I have done the first 2 but the third one for enabling it on the card I haven't, so I am very optimistic for tomorrow :D :D :D

Thank you again.

Post Reply