16:00:10 <dcavalca> #startmeeting Hyperscale SIG
16:00:10 <centbot> Meeting started Wed Aug 18 16:00:10 2021 UTC.  The chair is dcavalca. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:10 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:19 <dcavalca> #chair dcavalca jvreeland
16:00:19 <centbot> Current chairs: dcavalca jvreeland
16:00:23 <dcavalca> morning everyone
16:00:26 <jvreeland> morning
16:00:29 <bkircher> Hello there
16:01:16 <Eighth_Doctor> yo!
16:01:44 <dcavalca> I think we can get started
16:01:47 <dcavalca> #topic Followups
16:01:55 <dcavalca> did anybody have followups from the last round?
16:03:08 <dcavalca> guess not
16:03:11 <dcavalca> #topic Announcements
16:03:30 <dcavalca> osandov has published an updated btrfs-progs within the SIG
16:03:50 <Eighth_Doctor> I'm pushing new blivet for c8s-spin
16:04:01 <Eighth_Doctor> it has the fixes I submitted for the mdraid detection
16:04:10 <dcavalca> nice
16:05:19 <dcavalca> #info btrfs-progs-5.13.1 is now published in experimental
16:05:29 <bkircher> not sure if it's an announcment but I'd like to work on https://pagure.io/centos-sig-hyperscale/sig/issue/67
16:05:42 <dcavalca> bkircher: we'll do tickets in just a bit
16:06:06 <bkircher> 👍
16:06:09 <dcavalca> there's a few updates on the c9s front
16:06:15 <Eighth_Doctor> oh?
16:06:24 <dcavalca> looks like we should be getting a signed compose out today or tomorrow
16:06:38 <dcavalca> also, we should now be able to push c9s branches on git.c.o
16:06:57 <dcavalca> and we should have build tags soon
16:07:10 <dcavalca> so, things are finally moving :)
16:07:28 <Eighth_Doctor> yay
16:07:39 <Eighth_Doctor> are the c9s branches going to use the newer dist-git style instead of the older style?
16:07:53 <dcavalca> Eighth_Doctor: that is an excellent question, and I have no idea
16:08:25 <dcavalca> https://pagure.io/centos-infra/issue/336 is the ticket fyi
16:09:28 <dcavalca> my only other announcement is a reminder that we have the VC hangout later today
16:09:58 <Eighth_Doctor> new blivet build is coming: https://cbs.centos.org/koji/buildinfo?buildID=33837
16:10:15 <dcavalca> sweet
16:10:23 <dcavalca> anything else for announcements?
16:10:23 <bkircher> How would one join the VC hangout?
16:10:47 <dcavalca> bkircher: it's linked from https://www.centos.org/community/calendar/#Hyperscale_SIG_monthly_hangout
16:10:59 <bkircher> thx!
16:11:35 <dcavalca> alright, let's move on to
16:11:38 <dcavalca> #topic Tickets
16:11:41 <Eighth_Doctor> this is the correct URL: https://www.centos.org/community/calendar/#hyperscale-sig-monthly-hangout
16:12:00 <dcavalca> thanks Eighth_Doctor
16:12:17 <dcavalca> bkircher: wanna talk about #67 ?
16:12:30 <bkircher> would be delighted
16:13:10 <bkircher> so we're a very small CSP and did build our own qemu/libvirt/… packages o top of CentOS
16:13:56 <bkircher> we take directly from Fedora rawhide, test a bit, roll out, in case of regressions we backport from upstream libvirt and qemu
16:14:24 <Eighth_Doctor> so something to consider: the qemu packaging in RHEL is intended to be structured so that Fedora's qemu package can live in EPEL
16:14:46 <bkircher> and I thought it would be nice to have this within Stream one day… but would also like to have it in Stream 8
16:15:07 <Eighth_Doctor> https://bugzilla.redhat.com/show_bug.cgi?id=1974299#c5
16:15:16 <bkircher> Eighth_Doctor: indeed
16:15:30 <bkircher> however there are problems with other CentOS repos
16:15:37 <Eighth_Doctor> oh?
16:16:36 <bkircher> I think there are qemu packages shipping in "Advanced Virtualization" or some other repo that come with another qemu that created problems for us
16:16:48 <bkircher> but I think also for the EPEL one
16:17:28 <Eighth_Doctor> so Hyperscale policy is to prefer shipping in EPEL if it's possible
16:17:39 <Eighth_Doctor> and QEMU is certainly one of those where this may make sense, and it's worth investigating
16:17:52 <Eighth_Doctor> but I don't have a problem with us shipping an updated qemu if it turns out it can't go into EPEL
16:18:04 <dcavalca> yeah I hadn't realized doing QEMU in EPEL was a possiblity, that seems a better option if we can, as it would make it more widely available
16:18:28 <dcavalca> regardless of where this ends up living, my only request is to take care of https://pagure.io/centos-sig-hyperscale/sig/issue/55 while at it
16:18:32 <Eighth_Doctor> obviously, this doesn't apply to libvirt, and so we ship it in hyperscale
16:18:54 <Eighth_Doctor> dcavalca: yeah, there's also a couple of flags that I would like to have too
16:19:08 <bkircher> so EPEL would be the right place to look first… whats the reason behind libvirt there?
16:19:09 <Eighth_Doctor> specifically virgl and friends
16:19:27 <Eighth_Doctor> libvirt package name is used in RHEL, so EPEL can't ship it
16:19:44 <dcavalca> bkircher: EPEL cannot ship packages that already exist in RHEL
16:19:55 <bkircher> understand
16:19:56 <dcavalca> it works for qemu because the RHEL package is actually called qemu-kvm
16:20:12 <Eighth_Doctor> note that hyeprscale depends on epel anyway, so we would consume qemu from epel if it exists there
16:20:17 <Eighth_Doctor> *hyperscale
16:20:21 <dcavalca> yeah, I think that would work nicely
16:20:28 <Eighth_Doctor> that would also let me turn on more things in the hyperscale libvirt build too
16:21:05 <dcavalca> one thing worth pointing out is that the Hyperscale libvirt build is non-modular, as we can't do modules in CBS yet
16:21:10 <Eighth_Doctor> yeah
16:21:18 <dcavalca> this may or may not be a problem for you, but it's something to be aware of
16:21:31 <bkircher> sure
16:21:35 <bkircher> makes total sense
16:21:50 <Eighth_Doctor> we have a hotfix repo for this
16:21:56 <Eighth_Doctor> but it is something to be mindful of
16:22:09 <bkircher> we also have a couple of other packages (18 including with qemu and libvirt) around "virtualization" like edk2 and things
16:22:13 <Eighth_Doctor> as for qemu, since the intent is that qemu and qemu-kvm are non-conflicting, we should check if there's any accidental conflicts between the two
16:22:23 <bkircher> sure
16:22:27 <Eighth_Doctor> because those should be treated as bugs to fix
16:22:43 <Eighth_Doctor> at least for c9s
16:23:08 <Eighth_Doctor> if it turns out there are conflicts we can do qemu in hyperscale for c8s and work things out for c9s to ship in EPEL
16:23:36 <bkircher> ok, sounds reasonable
16:23:41 <dcavalca> I like this plan
16:24:23 <Eighth_Doctor> I can certainly help with qemu maintenance as I used to do it for $DAYJOB too
16:24:23 <dcavalca> bkircher: FYI, there's an EPEL meeting later today in case you need help from that crowd
16:24:36 <Eighth_Doctor> yeah, I'd recommend attending the EPEL meeting
16:24:45 <bkircher> thx, will look out for that
16:25:05 <dcavalca> bkircher: https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/S3SRO6OG54L24HJD2UT74ZBT53QLSGFR/
16:25:40 <Eighth_Doctor> speaking of EPEL
16:26:03 <Eighth_Doctor> tdawson is working on a new Plasma update, which will bring me to a point where I'll respin the media
16:26:21 <Eighth_Doctor> and arrfab set up the `spin-livecd` task for hyperscale, which I will then spend some time to test a scratch build in
16:26:44 <Eighth_Doctor> the Koji documentation confuses me on how it will work, but I guess I'll find out how well it'll actually work
16:28:30 <Eighth_Doctor> I still need to talk to bstinson about creating a way for livecd-tools to detect we're running in Koji to make it so modular hotfix packages work
16:28:44 <Eighth_Doctor> (non-modular packages overriding modular ones, basically)
16:29:08 <dcavalca> oh yeah we're going to need that
16:29:10 <Eighth_Doctor> I suspect this will be less of a problem for producing cloud images, though
16:29:15 <Eighth_Doctor> at least, I hope so
16:29:31 <Eighth_Doctor> fortunately, dcavalca and I already made appliance-tools handle btrfs fine
16:29:46 <Eighth_Doctor> but CBS not running on something that supports Btrfs is going to be a problem
16:29:51 <Eighth_Doctor> since we can't create and mount the filesystem
16:30:29 <dcavalca> oh that will be tricky for sure
16:31:00 <Eighth_Doctor> btrfs-fuse would be a solution here, or we're going to have to build a kmod-btrfs for CBS
16:31:52 <dcavalca> probably the latter, we're not going to have the fuse thing ready in time
16:32:04 <dcavalca> could also try to hack it via libguestfs
16:32:23 <Eighth_Doctor> has to work in mock :(
16:32:30 <Eighth_Doctor> that's where appliance-tools runs in
16:32:37 <Eighth_Doctor> but maybe, yeah
16:34:27 <dcavalca> do we have anything else for tickets?
16:34:32 <dcavalca> jvreeland: any update on the kernel side of things?
16:35:39 <jvreeland> looking at updating the one in experimental.  Haven't taken a look at where we stand on the signing stuff for real release so need to go through that again.
16:36:21 <jvreeland> It seemed to unlikely to happen soon, and people seemed insistent that we should have at least secure boot for release so not sure what the path forward would be.
16:36:57 <dcavalca> yeah we need to figure out a way forward there
16:37:53 <dcavalca> https://pagure.io/centos-infra/issue/307 doesn't have any update
16:40:11 <jvreeland> I think even if it resolved today it'd take a while for the infrastructure to get in place. So the real question seems to be when do we want to release a kernel by and is anyone planning on depending on it on some timeline.
16:40:40 <dcavalca> Eighth_Doctor: needs this kernel for the spins
16:40:44 <Eighth_Doctor> yeah
16:40:54 <Eighth_Doctor> and it might wind up being part of the solution for CBS too
16:40:56 <Eighth_Doctor> https://pagure.io/centos-sig-hyperscale/sig/issue/68
16:41:23 <dcavalca> I've just pinged the secure boot ticket
16:41:51 <dcavalca> though I agree with jvreeland, even if the politics of that get sorted out the implementation will still take a while, so we probably need to plan for a contingency here
16:43:26 <Eighth_Doctor> at least for now, I'm just saying "no uefi support" until we figure out what to do here
16:43:44 <Eighth_Doctor> but I also don't want to go beyond experimental without that
16:47:07 <jvreeland> nothings really change with the requirements or infra so it seems the resolution of that ticket is the still best way forward for uefi systems.
16:47:24 <dcavalca> yeah, let's see if we can get some movement there
16:47:50 <dcavalca> worst case, we can ship with a self signed key and instructions for how to enroll it
16:48:01 <dcavalca> which will make for a horrible user experience, but it's probably better than nothing
16:48:31 <Eighth_Doctor> yeah
16:48:50 <Eighth_Doctor> it also means that live ISOs will be crappy on UEFI :/
16:49:15 <dcavalca> unfortunately yes
16:49:22 <Eighth_Doctor> self signed keys means we need to ship our own grub too
16:49:30 <Eighth_Doctor> and shim
16:49:42 <Eighth_Doctor> that's required specifically for alternate key kernels
16:49:54 <dcavalca> ugh yeah that's going to suck
16:50:02 <Eighth_Doctor> since shim->grub->kernel path
16:50:27 <Eighth_Doctor> if we have to, we have to
16:50:50 <Eighth_Doctor> but at that point, we probably need to fork over money to get our own signing key
16:50:57 <Eighth_Doctor> from Microsoft
16:51:21 <dcavalca> how much money are we talking about?
16:51:31 <Eighth_Doctor> I think it's $99 a year?
16:51:48 <dcavalca> oh that's fine, I can probably have FB cover that
16:52:15 <jvreeland> a secureboot key costs less than a windows license?
16:52:29 <Eighth_Doctor> we'll need to design modified kernel packaging for this
16:52:31 <Eighth_Doctor> I'm double-checking right now
16:53:15 <Eighth_Doctor> https://mjg59.dreamwidth.org/12368.html?thread=419408
16:53:19 <Eighth_Doctor> it's a one-time $99 fee
16:53:38 <jvreeland> cool
16:53:59 <Eighth_Doctor> but if we go down this road, we're going to need to some special work for this
16:54:10 <dcavalca> yeah let's treat this as a last resort
16:54:35 <dcavalca> Eighth_Doctor: wanna add some details in that ticket to summarize this conversation and what going down this path would entail?
16:54:47 <Eighth_Doctor> sure
16:54:58 <dcavalca> this sounds awfully a lot like spinning up a shadow IT thing, and I'd really rather not do that if we can
16:54:58 <Eighth_Doctor> which ticket?
16:55:21 <Eighth_Doctor> yes
16:55:24 <Eighth_Doctor> we cannot build the packages in CBS if we do this
16:55:25 <dcavalca> https://pagure.io/centos-infra/issue/307
16:55:58 <Eighth_Doctor> the fundamental thing is that if we do this, we will need our own Koji system
16:55:58 <Eighth_Doctor> or our own OBS system
16:56:12 <dcavalca> I think we can make a decent case that this is barely reasonable for one SIG, and becomes pretty much insane if multiple SIGs need it
16:56:17 <Eighth_Doctor> because we need to be able to create builders that can sign
16:56:18 <dcavalca> and iirc kmods will need this too eventually
16:56:24 <Eighth_Doctor> yes
16:56:35 <Eighth_Doctor> they're doing self-signed thing
16:56:45 <Eighth_Doctor> but that works because they can have the key trusted by the kernel
16:56:52 <Eighth_Doctor> we need to boot, which makes this harder
16:56:59 <dcavalca> ah makes sense
16:57:09 <Eighth_Doctor> but fixing our problem will fix their problem too
16:57:24 <dcavalca> alright, we're almost out of time, lemme move to the next topic for the minutes
16:57:28 <dcavalca> #topic Membership
16:57:44 <dcavalca> bkircher: did you wanna join to help out with libvirt maintenance and stuff?
16:58:02 <bkircher> yes
16:58:16 <dcavalca> works for me
16:58:19 <dcavalca> any objections?
16:59:07 <Eighth_Doctor> none from me
16:59:14 <jvreeland> nope
16:59:47 <dcavalca> #info welcome bkircher to the SIG
16:59:59 <dcavalca> I'll take care of the formalities later today
17:00:08 <dcavalca> #topic Misc
17:00:23 <dcavalca> we are out of time folks, have a good one!
17:00:40 <dcavalca> #endmeeting