Page 1 of 1

Xen DomU's randomly freezing

Posted: 2018/04/18 09:57:19
by deltadelta
Hi all,

We've been in the process of migrating all our hypervisors over to CentOS 7 using Xen. Once we had a few up and running we started to notice that the DomU's would randomly freeze. They become unresponsive to any network traffic, stop consuming CPU resources on the hypervisor and it's not possible to log in to the console locally using:

Code: Select all

virsh console <domain>

We can get as far as typing a username and hitting return, but the DomU just hangs there. It doesn't seem to matter what Linux distro the DomU is running, it affects them all. The only way we can get them back is by destroying and recreating them (far from ideal!).

After a bit of research and digging around, we eventually found these 2 nuggets:
https://wiki.gentoo.org/wiki/Xen#Xen_do ... nel_4.3.2B
https://www.novell.com/support/kb/doc.php?id=7018590

They both advise adding the command line argument:

Code: Select all

gnttab_max_frames=256
(the default is 32).
We applied this change and all hypervisors rand stable for around a week until DomU's started freezing again (we've since tried even higher values, to no avail). More research later led me to https://bugs.centos.org/view.php?id=14258 and https://bugs.centos.org/view.php?id=14284 (which are essentially the same report). There hasn't really been any movement on these tickets so I thought I'd see if the forum could reach a wider audience. Have any others had issues with Xen and DomU's locking up in CentOS 7?

Any help would be much appreciated and would mean my Ops guys could get a decent sleep!
Cheers

Re: Xen DomU's randomly freezing

Posted: 2018/04/18 12:37:24
by TrevorH
Xen4CentOS is produced by one of the SIGs and as far as I know, none of them have even heard of the forums let alone read them :-(

I'd recommend writing to the centos-devel mailing list as the SIG members are more likely to be a part of that than here.

Re: Xen DomU's randomly freezing

Posted: 2018/04/20 14:16:18
by deltadelta
Thanks TrevorH, I've sent all the details I have onto the mailing list. I'll post back here with any updates in case anyone else has the same issues)

Re: Xen DomU's randomly freezing

Posted: 2018/06/19 11:19:25
by deltadelta
Just an update on this problem. We eventually upgraded Xen to 4.8 and changed the kernel that the VMs use to the LTS kernel from elrepo. Since doing this, we haven't had any lock ups or freezes in over a month. It might be that the 4.8 upgrade is enough, but we tried a newer kernel first as the initial attempt to update to 4.8 failed.