16:00:09 <dcavalca> #startmeeting Hyperscale SIG
16:00:09 <centbot> Meeting started Wed Jul  7 16:00:09 2021 UTC.  The chair is dcavalca. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:17 <michel-slm> no lag today, yay
16:00:17 <dcavalca> #chair dcavalca jvreeland
16:00:17 <centbot> Current chairs: dcavalca jvreeland
16:00:21 <dcavalca> morning everyone
16:00:23 <Eighth_Doctor> yup
16:00:27 <Eighth_Doctor> hey y'all
16:00:29 <anitazha> 'morning
16:00:30 <michel-slm> ohayo
16:00:32 <dcavalca> we'll get started in a minute
16:01:45 <dcavalca> #topic Followups
16:02:19 <dcavalca> the issues we'd found with composes.centos.org have been sorted out, so it can now be mirrored safely
16:02:38 <dcavalca> and the FB mirror at http://mirror.facebook.net/centos-composes/8/ should catch up soon
16:02:57 <michel> 👍️
16:02:58 <dcavalca> jvreeland: any news on the devconf.us talk?
16:03:36 <jvreeland> nope, i don't think we find out till august
16:03:44 <dcavalca> alright
16:03:49 <dcavalca> anitazha: any update on the new systemd release?
16:05:10 <anitazha> you mean 249? it was going to get tagged today or tomorrow but another regression might have appeared
16:05:31 <dcavalca> yeah, I just saw it's still at rc, I thought we'd have the final by now
16:06:04 <Eighth_Doctor> there's one last PR to merge: https://github.com/systemd/systemd/pull/20154
16:06:15 <dcavalca> also, the very last one failed to build in CI, but I think it's some infra issue: https://cbs.centos.org/koji/taskinfo?taskID=2499954
16:06:23 <dcavalca> as it failed unpacking the tarball, which is definitely weird
16:06:34 <Eighth_Doctor> that PR is the last thing for 249 milestone
16:07:09 <dcavalca> yeah, should be due soon either way
16:07:15 <anitazha> yup
16:07:36 <dcavalca> alright, I think that's all for followups
16:07:40 <dcavalca> #topic Announcements
16:07:56 <dcavalca> #info SIG quarterly report is out: https://blog.centos.org/2021/06/centos-hyperscale-sig-quarterly-report-for-2021q2/
16:08:06 <dcavalca> thanks everyone for the help putting this together
16:09:07 <dcavalca> #info missing -devel packages and EPEL are now available on our CBS tags
16:09:19 <Eighth_Doctor> woot
16:09:26 <dcavalca> that's courtesy of arrfab, and it should make our life a lot easier
16:09:38 <Eighth_Doctor> that means we now have access to livecd-tools and appliance-tools in CBS
16:09:40 * arrfab heard his name quoted here ...
16:09:52 <Eighth_Doctor> which means we can build images using spin-livecd and createAppliance
16:10:21 <Eighth_Doctor> (both of these are maintained in EPEL, so having EPEL sourced means those commands will work)
16:10:39 <dcavalca> it'll also simplify the util-linux rebuild we have, I'll try and find some time to cut another one so we can drop our internal libuser and openldap
16:10:47 <anitazha> nice
16:11:08 <dcavalca> #info LLVM 12 is now available in c8s
16:11:25 <dcavalca> so I'll probably untag our non-modular version as soon as I've validated it actually works
16:11:51 <dcavalca> #info outstanding libvirt bug: https://pagure.io/centos-sig-hyperscale/package-bugs/issue/2
16:12:04 <dcavalca> signal boosting this as we hit it in production
16:12:15 <dcavalca> it affects both c8s proper and our Hyperscale backport
16:12:46 <dcavalca> anybody else has announcements?
16:13:40 <dcavalca> alright, let's move on to
16:13:45 <dcavalca> #topic Tickets
16:14:04 <dcavalca> I think we can finally close #47 now that we have EPEL working in our tags
16:14:42 <dcavalca> we do still need to come up with a good solution for #59 (centos-release-hyperscale should enable powertools repo on CentOS Stream)
16:14:50 <dcavalca> michel: iirc you had a prototype for this?
16:15:13 <michel> dcavalca: yes
16:16:00 <michel> carlwgeorge hated it, so it's probably not a good idea. the longer term solution is to make it possible to overlay DNF configuration files (similar to how systemd does it)
16:16:18 <michel> TL;DR the prototype involves doing dnf config-manager --enable in %posttrans
16:16:29 <michel> s/dnf config-manager/sedding the repo to be enabled
16:16:43 <dcavalca> we might need that as a stopgap solution still, as getting DNF to support overlays will probably take a while
16:17:37 <carlwgeorge> would it be that bad to just document that powertools needs to be enabled as part of the initial setup?  that's what epel does.
16:18:01 <Eighth_Doctor> yes
16:18:26 <Eighth_Doctor> and it's bad for epel too
16:18:30 <Eighth_Doctor> I still regularly get bug reports about it
16:18:31 <dcavalca> I'd rather minimize the amount of manual steps users have to take if we can
16:19:21 * carlwgeorge shrugs
16:19:39 <michel> <Eighth_Doctor "I still regularly get bug report"> me too
16:20:06 <Eighth_Doctor> for the live media builds, I'm forcing powertools on: https://pagure.io/centos-sig-hyperscale/livemedia-kickstarts/blob/c8s/f/centos-stream-hyperscale-common.ks#_16
16:20:19 <michel> for the Hyperscale image, I think we can just do that in kickstart
16:20:20 <carlwgeorge> fwiw, i suggested for 9 making powertools enabled=1 but in an optional subpackage, so other release packages could require it seamlessly, but bstinson shot it down
16:20:21 <michel> yeah
16:20:33 <michel> carlwgeorge: sigh :(
16:20:38 * Eighth_Doctor sighs
16:20:41 <michel> what was the rationale?
16:20:44 <Eighth_Doctor> we can bring it up in the Stream meeting again
16:20:50 <Eighth_Doctor> I don't think we've brought that up recently there
16:20:53 <michel> I like having two repo definitions, one with enabled=1 and one with enabled=0
16:21:14 <michel> but generated from the same source RPM so we don't accidentally diverge
16:21:33 <michel> the problem with force enabling ourselves is... if that definition ever changes, DNF will never overwrite the changed repo file :(
16:21:43 <Eighth_Doctor> yeah
16:21:48 <Eighth_Doctor> I'm not happy about it
16:21:53 <Eighth_Doctor> but what else to do?
16:22:35 <michel> let's bring this up to bstinson ourselves next week
16:22:39 <dcavalca> yeah, I don't really see a way out either
16:22:41 <dcavalca> agreed
16:22:48 <michel> I like carl's suggestion as an interim stopgap
16:22:56 <dcavalca> this will make a good office hours discussion topic
16:22:57 <carlwgeorge> he likes the symmetry between `subscription-mananger enable` and `dnf config-manager enable` (or whatever the flags/syntax is)
16:23:54 <dcavalca> ... is subscription-manager even a thing with centos? I thought it was RHEL specific
16:24:17 <michel> I think it was a thing if you have your own Satellite instance?
16:24:36 <dcavalca> oh, right
16:24:41 <dcavalca> anyway, let's move on
16:24:41 <michel> so.. yeah, in the future once we can do dnf config overlays, config-manager enable will do exactly that
16:24:58 <dcavalca> jvreeland: any update on the kernel stuff for #7 ?
16:25:39 <Eighth_Doctor> oddly enough, I kinda have an update
16:26:09 <dcavalca> please do share :)
16:26:10 <Eighth_Doctor> jforbes reached out to me earlier today and let me know that the CKI/ARK folks have made progress on supporting overlay configs for SIGs
16:26:10 <jvreeland> none here, it's in experimental give it a try i guess
16:26:19 <jvreeland> oh nice
16:26:29 <dcavalca> that's great
16:26:48 <Eighth_Doctor> Don Zickus apparently got that going a bit ago
16:27:03 <Eighth_Doctor> I asked him to send us an email about it on the thread we're on about it so we can get everyone in sync and go through the next steps
16:27:21 <dcavalca> perfect, thanks
16:27:29 <Eighth_Doctor> it seems like they are interested in having us work in their tree for the Hyperscale kernel, and I'm hopeful we can make progress to make that a reality
16:27:53 <dcavalca> yeah, I think that'll work best for everybody if we can get the workflow down
16:28:15 <Eighth_Doctor> that will also make it so there's a path to on-ramping features from us into RHEL itself
16:28:24 <dcavalca> yup
16:28:31 <michel> speaking of that... there was no important emails in the past week or so right?
16:28:39 <michel> we had a work email outage last week
16:28:48 <Eighth_Doctor> no, don't think so?
16:29:01 <Eighth_Doctor> other than pystemd FTBFS in Fedora
16:29:30 <dcavalca> Eighth_Doctor: saw that one, I'm asking Alvaro to take a look from our side
16:29:37 <Eighth_Doctor> awesome
16:29:57 <dcavalca> ok, let's move on to
16:30:08 <dcavalca> #topic Membership
16:30:23 <dcavalca> we have a membership request from cmurf in #63
16:30:31 <dcavalca> cmurf[m]: wanna introduce yourself to the group?
16:31:07 * michel waves
16:31:44 <anitazha> hello cmurf[m] !
16:32:35 <Eighth_Doctor> hey cmurf!
16:34:24 <dcavalca> I suspect the matrix bridge is playing tricks again
16:34:39 <dcavalca> for those who haven't worked with cmurf before, he's been one of the main drivers for btrfs in Fedora
16:35:11 <michel-slm> test libera->matrix
16:35:22 <michel> works for me both ways, but cmurf has had issues
16:35:47 <dcavalca> yeah, it's not been terribly reliable
16:37:06 <dcavalca> let's move on and we can go back to this later if it gets sorted out, or finish in the ticket if needed
16:37:13 <dcavalca> #topic Misc
16:37:21 <dcavalca> the only other thing I had was the secure boot stuff
16:37:50 <dcavalca> https://pagure.io/centos-infra/issue/307
16:38:16 <dcavalca> there's been some more discussion here, and I think we've landed on a compromise solution that might be acceptable for RH, but we're still waiting to hear back
16:39:39 <Eighth_Doctor> I wonder if we're going to be forced to maintain shim and grub2
16:39:46 <dcavalca> hopefully not
16:39:57 <dcavalca> but this is another good topic for office hours
16:41:02 <dcavalca> Eighth_Doctor: jvreeland: did you manage to get your access to the openshift CI sorted out?
16:41:33 <dcavalca> I haven't had time to work on that lately, but hoping to pick it back up later this month as I still need to automate the container builds
16:41:46 <Eighth_Doctor> I can log in and see the hyperscale namespace
16:41:53 <jvreeland> Eighth_Doctor: I did, i've got the repo syncing kernel stuff working I think, haven't checked since i redeployed it tbh.
16:42:22 <dcavalca> great, thanks
16:42:32 <Eighth_Doctor> do we have any stuff for how we deploy into it?
16:43:23 <dcavalca> Eighth_Doctor: not really, I have the systemd stuff up at https://pagure.io/centos-sig-hyperscale/systemd-releng which is a good start
16:43:30 <Eighth_Doctor> 👍️
16:43:37 <dcavalca> I meant to generalize this and write some docs, but haven't gotten around it yet
16:44:09 <dcavalca> but it's a pretty standard openshift setup, so if you're used to that you should be fine
16:44:31 <dcavalca> there's some additional stuff we can get enabled if we need jenkins to deploy to worker nodes for testing, but I haven't looked into that yet
16:44:38 <dcavalca> and there's an outstanding ticket to get VMs enabled
16:45:02 <dcavalca> https://pagure.io/centos-infra/issue/308
16:45:10 <Eighth_Doctor> ah
16:46:18 <jvreeland> dcavalca: I didn't see the hyperscale bot as a member for our pagure group so the ssh keys are on my profile for testing i'd rather they not be though. Did i just miss something?
16:47:20 <dcavalca> jvreeland: the bot has its own credentials and stuff in the secrets repo
16:47:35 <dcavalca> if you need per-repo keys to do stuff, you can generate those from the pagure repo settings
16:47:44 <dcavalca> just make sure you add them to the secrets repo as well so they don't get lost
16:47:55 <dcavalca> that's what I did for systemd to be able to push to the repo we mirror
16:48:08 <dcavalca> I think it's called "deployment key" in pagure
16:48:29 <jvreeland> ahh that's what I was missing then, thank you.
16:49:43 <dcavalca> anything else folks want to discuss?
16:49:46 <jvreeland> unrelated to the kernel ticket, I thought someone mentioned a backport for iptables at some point?
16:49:53 <dcavalca> jvreeland: yes, that was me
16:50:09 <dcavalca> I still need to clean up our internal one and publish it
16:50:09 <jvreeland> Is that still the plan I just hadn't heard anything in a bit
16:50:14 <jvreeland> cool
16:50:18 <dcavalca> I did publish arptables and ebtables
16:50:52 <dcavalca> I can try and get that sorted out later this month
16:50:53 <Eighth_Doctor> I'll be updating anaconda and respinning the live media this week
16:51:08 <Eighth_Doctor> the centos-logos update finally was released in yesterday's compose
16:51:14 <dcavalca> sweet
16:51:20 <Eighth_Doctor> and carlwgeorge added a patch to anaconda to fix branding stuff
16:51:26 <Eighth_Doctor> so I need to pull that into our tree
16:52:16 <Eighth_Doctor> did we get a dnf + rpm stack update?
16:52:27 <dcavalca> I know malmond updated librepo
16:52:33 <Eighth_Doctor> because I think last I checked, we were out of date compared to c8s
16:52:40 <dcavalca> not sure about the rest of the stack, lemme check
16:53:17 <michel> unrelated, Conan Kudo does the new Emacs work for you?
16:53:26 <Eighth_Doctor> haven't tried yet
16:53:28 <dcavalca> dnf-4.4.2-11.3.hsx.el8 and rpm-4.14.3-13.1.hsx.el8 are the last ones we have
16:53:32 <Eighth_Doctor> I was going to this week as part of respinning the media
16:53:43 <Eighth_Doctor> (emacs is getting included because I can :D)
16:54:17 <Eighth_Doctor> if it's good enough for macOS it's good enough for CentOS :D
16:54:36 <dcavalca> Eighth_Doctor: yeah, those are the last versions, I checked internally as well to be sure
16:55:05 <Eighth_Doctor> we'll need the latest stuff pushed by RHEL merged in
16:55:06 <dcavalca> we do have librepo-1.14.0-1.1.hsx.el8 though
16:55:14 <Eighth_Doctor> otherwise we'll get a broken image build at the end of the week
16:55:26 <Eighth_Doctor> (when I go make it, that is)
16:56:03 <dcavalca> malmond is out, but maybe michel and I can give this a try
16:56:11 <Eighth_Doctor> that'd be great :)
16:56:12 <dcavalca> assuming the patches still apply cleanly it shouldn't be too bad
16:57:13 <michel> dcavalca: what do I need to try?
16:57:43 <Eighth_Doctor> Michel Alexandre Salim: rebasing the RPM+DNF stack (rpm, librepo, libdnf, dnf) with the latest stuff from RHEL
16:58:05 <dcavalca> michel: not sure, I don't think malmond documented his process for this, but we'll figure it out
16:58:20 <dcavalca> alright, we're almost out of time
16:59:25 <dcavalca> looks like cmurf is still MIA, so we'll continue in the ticket there
16:59:30 <dcavalca> thanks everyone!
16:59:41 <dcavalca> #endmeeting