Custom Kernel - Packaging kernel with vfio-pci support

Issues related to applications and software problems
supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Custom Kernel - Packaging kernel with vfio-pci support

Post by supmethods » 2015/01/08 13:08:47

Hi

I'm currently using CentOS 6 as a KVM host and will need to customise the kernel to suppport vfio-pci.

Where can I get the spec file for kernel 3.10+ or is there a src.rpm I can download that'll support this feature?

Additionally, I'll also need to patch the kernel with acs override patch and i915 vga arbiter fixes as mentioned in this article: https://bbs.archlinux.org/viewtopic.php?id=162768

Here's a thread for the ACS override patch: https://lkml.org/lkml/2013/5/30/513
Here's a thread for the i915 vga arbiter fix: https://lkml.org/lkml/2014/5/9/517

How to I apply these patches to the kernel? Thanks.
Last edited by supmethods on 2015/01/08 13:46:55, edited 1 time in total.

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by toracat » 2015/01/08 13:28:21

For kernel 3.18, look in ELRepo's kernel-ml.

For adding patches, look in CentOS wiki article on How to build custom kernel.
CentOS Forum FAQ

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by supmethods » 2015/02/15 12:46:52

It appears ELRepo only provides only RPM kernels. I'm looking for SRPMs, I'd can customise the kernel without having to write the SPECs file from scratch.

Kernel.org doesn't seem to offer srpm and only the source codes. Is there any other places I can get the SRPM. Thank you.

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by toracat » 2015/02/15 13:47:13

SRPMs are available. For example, if you need the ones for kernel-ml/lt for EL6, look in:

http://elrepo.org/linux/kernel/el6/SRPMS/

They are "nosrc" packages. Just download a matching tar file from kernel.org to make it complete.
CentOS Forum FAQ

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by supmethods » 2015/02/16 13:00:35

Ok thanks. I'm a bit confused with the patching though. From the article on patching, it mentions to specfy the patch files in the spec files.

However, I'm looking at this link here: https://lkml.org/lkml/2013/5/30/513 which appears to be the result of diff. The diff comparison should add/remove entries required for patching the kernel.

In the thread mentioned, it looks like some part of the message should go into a patchfile but not all of it.

My questions are as below:
* Is there a place where I can download these patchfiles?
* Will there be issues with this patchfiles if I'm using a kernel with updates done to it? I believe the patch in the thread is for kernel 3.18.5. I'm looking to patch up kernel 3.18.6.
* Or should I create two patch files as below?

Patchfile 1:

Code: Select all

--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2349,6 +2349,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 		nomsi	Do not use MSI for native PCIe PME signaling (this makes
 			all PCIe root ports use INTx for all services).
 
+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex
+
 	pcmv=		[HW,PCMCIA] BadgePAD 4
 
 	pd.		[PARIDE]
Patchfile 2:

Code: Select all

--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3292,11 +3292,113 @@ struct pci_dev *pci_get_dma_source(struct pci_dev *dev)
 	return pci_dev_get(dev);
 }
 
+static bool acs_on_downstream;
+static bool acs_on_multifunction;
+
+#define NUM_ACS_IDS 16
+struct acs_on_id {
+	unsigned short vendor;
+	unsigned short device;
+};
+static struct acs_on_id acs_on_ids[NUM_ACS_IDS];
+static u8 max_acs_id;
+
+static __init int pcie_acs_override_setup(char *p)
+{
+	if (!p)
+		return -EINVAL;
+
+	while (*p) {
+		if (!strncmp(p, "downstream", 10))
+			acs_on_downstream = true;
+		if (!strncmp(p, "multifunction", 13))
+			acs_on_multifunction = true;
+		if (!strncmp(p, "id:", 3)) {
+			char opt[5];
+			int ret;
+			long val;
+
+			if (max_acs_id >= NUM_ACS_IDS - 1) {
+				pr_warn("Out of PCIe ACS override slots (%d)\n",
+					NUM_ACS_IDS);
+				goto next;
+			}
+
+			p += 3;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].vendor = val;
+
+			p += strcspn(p, ":");
+			if (*p != ':') {
+				pr_warn("PCIe ACS invalid ID\n");
+				goto next;
+			}
+
+			p++;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].device = val;
+			max_acs_id++;
+		}
+next:
+		p += strcspn(p, ",");
+		if (*p == ',')
+			p++;
+	}
+
+	if (acs_on_downstream || acs_on_multifunction || max_acs_id)
+		pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n");
+
+	return 0;
+}
+early_param("pcie_acs_override", pcie_acs_override_setup);
+
+static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags)
+{
+	int i;
+
+	/* Never override ACS for legacy devices or devices with ACS caps */
+	if (!pci_is_pcie(dev) ||
+	    pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS))
+		return -ENOTTY;
+
+	for (i = 0; i < max_acs_id; i++)
+		if (acs_on_ids[i].vendor == dev->vendor &&
+		    acs_on_ids[i].device == dev->device)
+			return 1;
+
+	switch (pci_pcie_type(dev)) {
+	case PCI_EXP_TYPE_DOWNSTREAM:
+	case PCI_EXP_TYPE_ROOT_PORT:
+		if (acs_on_downstream)
+			return 1;
+		break;
+	case PCI_EXP_TYPE_ENDPOINT:
+	case PCI_EXP_TYPE_UPSTREAM:
+	case PCI_EXP_TYPE_LEG_END:
+	case PCI_EXP_TYPE_RC_END:
+		if (acs_on_multifunction && dev->multifunction)
+			return 1;
+	}
+
+	return -ENOTTY;
+}
+
 static const struct pci_dev_acs_enabled {
 	u16 vendor;
 	u16 device;
 	int (*acs_enabled)(struct pci_dev *dev, u16 acs_flags);
 } pci_dev_acs_enabled[] = {
+	{ PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides },
 	{ 0 }
 };
Thank you.

User avatar
AlanBartlett
Forum Moderator
Posts: 9345
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by AlanBartlett » 2015/02/16 21:11:13

Having some understanding of the subject, I'll attempt to answer your three questions . . .
* Is there a place where I can download these patchfiles?
* Will there be issues with this patchfiles if I'm using a kernel with updates done to it? I believe the patch in the thread is for kernel 3.18.5. I'm looking to patch up kernel 3.18.6.
* Or should I create two patch files as below?
* The patch files should be placed in your rpmbuild/SOURCES/ directory.
* There might be. Either the patches will apply cleanly or will fail. It is a question of try it and see. (You really should not be using either the 3.18.5 or 3.18.6 sources, as both have been deprecated. The current supported release in the 3.18.X branch is 3.18.7)
* The choice is yours. I prefer separate, individual, patches as it simplifies things in the case when a patch does not apply cleanly.

To help you get started, I have taken the kernel-ml-3.18.6-1.el6.elrepo.nosrc.rpm file and edited it so that it will:

(1) use the 3.18.7 source tarball.
(2) apply the two separate patch files (as discussed above).

You can download the newly created kernel-ml-3.18.7-1.el6.elrepo.nosrc.rpm file from the http://elrepo.org/people/ajb/tmp/ directory. A comparison of its contents with that of the kernel-ml-3.18.6-1.el6.elrepo.nosrc.rpm file should be a sufficient example of the techniques used.
Image 100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by supmethods » 2015/02/17 13:49:56

Thank you. The compilation process has begun, I'm just waiting for it to finish.

I've got one other question. Normally when I want to enable/disable a kernel module, I usually make modification to the the config files in the SOURCE directory.

Is this what I normally should do? Or should I make the modification in the buildtree? Thanks.

User avatar
AlanBartlett
Forum Moderator
Posts: 9345
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by AlanBartlett » 2015/02/17 16:40:41

To clarify, I did not apply the i915 VGA arbiter fix patch but just the ACS override, split into two patches. Therefore you will need to apply the i915 VGA arbiter fix patch . . .

If you wish to change the kernel configuration, the process is as follows --
  • Install the nosrc.rpm file into your ~/rpmbuild/ tree.
  • Copy the correct tarball (obtained from the Linux Kernel Archive) to the ~/rpmbuild/SOURCES/ directory.
  • From the ~/rpmbuild/SPECS/ directory execute rpmbuild -bp *.spec
  • Go to the ~/rpmbuild/BUILD/ker*/lin*/ directory.
  • Copy the relevant configuration file (32-bit PAE, 32-bit NONPAE or 64-bit) to .config
  • Execute make oldconfig
  • Execute make menuconfig
  • Copy the newly updated .config file back to the relevant configuration file (in the current directory).
  • Copy the updated configuration file to the ~/rpmbuild/SOURCES/ directory.
  • Return to the ~/rpmbuild/SPECS/ directory and then start the rpmbuild -ba process with any relevant arguments.
Image 100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by supmethods » 2015/03/07 05:39:26

Thank you for the assistance. Installed and compiled the kernel but QEMU will also be required to be compiled for vfio feature.

I've decided to move over Fedora as I feels it's more appropriate for the system I'm building right now.

Thanks for guiding me, it's knbowledge I'll carry with me for the many years to come.

User avatar
AlanBartlett
Forum Moderator
Posts: 9345
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: Custom Kernel - Packaging kernel with vfio-pci support

Post by AlanBartlett » 2015/03/07 16:18:34

You are welcome. :)
Image 100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Post Reply