Xen DomU's randomly freezing

General support questions
Post Reply
deltadelta
Posts: 5
Joined: 2015/09/30 16:14:43

Xen DomU's randomly freezing

Post by deltadelta » 2018/04/18 09:57:19

Hi all,

We've been in the process of migrating all our hypervisors over to CentOS 7 using Xen. Once we had a few up and running we started to notice that the DomU's would randomly freeze. They become unresponsive to any network traffic, stop consuming CPU resources on the hypervisor and it's not possible to log in to the console locally using:

Code: Select all

virsh console <domain>

We can get as far as typing a username and hitting return, but the DomU just hangs there. It doesn't seem to matter what Linux distro the DomU is running, it affects them all. The only way we can get them back is by destroying and recreating them (far from ideal!).

After a bit of research and digging around, we eventually found these 2 nuggets:
https://wiki.gentoo.org/wiki/Xen#Xen_do ... nel_4.3.2B
https://www.novell.com/support/kb/doc.php?id=7018590

They both advise adding the command line argument:

Code: Select all

gnttab_max_frames=256
(the default is 32).
We applied this change and all hypervisors rand stable for around a week until DomU's started freezing again (we've since tried even higher values, to no avail). More research later led me to https://bugs.centos.org/view.php?id=14258 and https://bugs.centos.org/view.php?id=14284 (which are essentially the same report). There hasn't really been any movement on these tickets so I thought I'd see if the forum could reach a wider audience. Have any others had issues with Xen and DomU's locking up in CentOS 7?

Any help would be much appreciated and would mean my Ops guys could get a decent sleep!
Cheers

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Xen DomU's randomly freezing

Post by TrevorH » 2018/04/18 12:37:24

Xen4CentOS is produced by one of the SIGs and as far as I know, none of them have even heard of the forums let alone read them :-(

I'd recommend writing to the centos-devel mailing list as the SIG members are more likely to be a part of that than here.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

deltadelta
Posts: 5
Joined: 2015/09/30 16:14:43

Re: Xen DomU's randomly freezing

Post by deltadelta » 2018/04/20 14:16:18

Thanks TrevorH, I've sent all the details I have onto the mailing list. I'll post back here with any updates in case anyone else has the same issues)

deltadelta
Posts: 5
Joined: 2015/09/30 16:14:43

Re: Xen DomU's randomly freezing

Post by deltadelta » 2018/06/19 11:19:25

Just an update on this problem. We eventually upgraded Xen to 4.8 and changed the kernel that the VMs use to the LTS kernel from elrepo. Since doing this, we haven't had any lock ups or freezes in over a month. It might be that the 4.8 upgrade is enough, but we tried a newer kernel first as the initial attempt to update to 4.8 failed.

Post Reply