16:00:52 <dcavalca> #startmeeting Hyperscale SIG
16:00:52 <centbot> Meeting started Wed Mar  2 16:00:52 2022 UTC.  The chair is dcavalca. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:52 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:01:03 <dcavalca> #chair dcavalca jvreeland
16:01:03 <centbot> Current chairs: dcavalca jvreeland
16:01:06 <dcavalca> morning folks
16:01:09 <michel> Hi!
16:01:16 <daandemeyer> Hi
16:01:17 <aekoroglu> Hello
16:01:19 <oidoming> Hi
16:03:15 <dcavalca> let's get started
16:03:23 <dcavalca> #topic Followups
16:04:05 <dcavalca> I have one followup on the openshift access for our CI
16:04:28 <dcavalca> if you need access, you should add your email to https://pagure.io/centos-infra/issue/639 so they can add you to the appropriate openshift project
16:04:35 <dcavalca> davdunc ^^ specifically
16:04:35 <davdunc> .hello davdunc
16:04:45 <anitazha> hello
16:05:10 <dcavalca> daandemeyer: wanna talk about the systemd update?
16:05:16 <daandemeyer> Yeah
16:05:39 <daandemeyer> So this past week I've done a few changes to the systemd RPM process for the Hyperscale branches
16:05:58 <daandemeyer> To summarize, we switched to the dist-git flat layout that's used by the Fedora RPM repo as well
16:06:10 <daandemeyer> This allows us to use git to sync changes from upstream instead of doing it manually
16:06:31 <daandemeyer> On top of that, we now use tarballs from the pagure staging repo instead of tarballs from systemd-stable on github
16:06:51 <daandemeyer> This saves us from having to generate patch files for all of the commits we backport to the staging repo
16:07:12 <daandemeyer> I have a PR open on the sig pagure repo updating the documentation to reflect these changes
16:07:31 <michel> Nice!
16:07:33 <anitazha> <3 this is literally my dream come true
16:08:10 <daandemeyer> That's more or less it, there's still an issue with c9s builders that I'm waiting to be resolved before we can do builds for the Stream 9 RPM branch
16:08:37 <daandemeyer> https://pagure.io/centos-infra/issue/678
16:09:17 <davdunc> fantastic effort!
16:09:53 <dcavalca> once the infra ticket is sorted out, we should look into converting more of our packages to the flat layout
16:10:02 <dcavalca> as it'll make it a lot easier to minimize the delta with Fedora
16:10:27 <dcavalca> thanks daandemeyer
16:11:02 <dcavalca> chantra: wanna talk about rpm CoW?
16:11:35 <chantra__> yeah, well RPM CoW proper, not many changes except that we have it deployed "widely" internally so I am fairly confident the current patch is working
16:11:47 <chantra__> both when CoW is enabled, or when it is not.
16:12:23 <chantra__> I did work on porting this to c9s, I actually have the patch backported on my repo/branch, but we cannot build for c9s hsx yet.
16:12:45 <chantra__> I picked up on @Eighth_Doctor 's PR to enable c9s hsx mocks https://bugzilla.redhat.com/show_bug.cgi?id=2059424
16:13:04 <chantra__> went down the rabbit hole and we are currently blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2059424
16:13:32 <dcavalca> bstinson ^^ fyi
16:13:41 <chantra__> there is a few approaches to fix this, the lowest hanging fruit would seem to be to generate a new compliant key
16:14:28 <chantra__> for now, in term of local build with mock, I unblocked myself with using libgcrypt instead of openssl, built the rpm, copied and installed them in the bootstrap mock
16:14:57 <chantra__> https://gitlab.com/redhat/centos-stream/rpms/rpm/-/merge_requests/16 isused to build the new RPM
16:15:18 <chantra__> this unblock only local build, currently cannot build on koji though.
16:15:24 <chantra__> that's it for me
16:15:41 <dcavalca> thanks chantra
16:15:47 <dcavalca> anybody else has followups?
16:16:49 <michel> Tangential followup
16:17:00 <kcwells71> Not a followup, but is there a separate section for opens?
16:17:16 <michel> We have some new packages in EPEL 8 Next now that RH has updated gflags
16:17:29 <dcavalca> kcwells71: we usually do open floor at the end
16:17:40 <kcwells71> dcavalca: gotcha
16:17:58 <michel> And that means if we have any HS package that uses gflags or glog we might want to rebuild them
16:18:40 <dcavalca> michel: good point
16:18:40 <Eighth_Doctor> 👋
16:18:46 <dcavalca> I think long term this feeds into https://pagure.io/centos-sig-hyperscale/sig/issue/54
16:19:17 <dcavalca> i.e. once we have automation in place to flag rebuilds of our packages, extending that to cover revdeps shouldn't be too hard
16:19:33 <dcavalca> for now, we can probably do a manual query and do this by hand
16:19:42 <Eighth_Doctor> also, we probably need to do rebuilds of anything that wasn't built before we started having a Vendor tag set
16:19:48 <Eighth_Doctor> err wasn't built after
16:20:24 <dcavalca> michel: Eighth_Doctor: mind filing tickets for these so we don't forget?
16:20:26 <Eighth_Doctor> I'm probably soon going to branch the rpm and dnf stack to include my base changes for c9s
16:20:47 <michel> rpmdev-newspec should be auto spec aware so we can always call it the same way on all packages right?
16:20:50 <Eighth_Doctor> I don't need rpmcow yet, but I definitely need to change several defaults for producing the workstation image
16:20:56 <dcavalca> Eighth_Doctor: please coordinate with chantra on that so you don't accidentally step onto each other
16:21:08 <Eighth_Doctor> please don't use rpmautospec in cbs packages, I'm pretty sure that won't work properly
16:21:42 <dcavalca> oh yeah I doubt that works, especially for C8
16:22:31 <dcavalca> let's go to
16:22:35 <dcavalca> #topic Announcements
16:22:52 <dcavalca> there's EPEL office hours right after this if anybody is interested: https://communityblog.fedoraproject.org/epel-office-hours/
16:23:12 <michel> Eighth_Doctor: Oh, yeah, I mean we can use the same bump and rebuild logic as on fedora
16:24:39 <dcavalca> anything else for announcements?
16:25:44 <dcavalca> k, moving on
16:25:47 <dcavalca> #topic Tickets
16:26:21 * dcavalca patiently waits for pagure to load
16:26:38 <davdunc> it's the worst part of running a meeting.
16:27:01 <dcavalca> Eighth_Doctor: you had https://pagure.io/centos-sig-hyperscale/sig/issue/99 tagged for meeting a while ago, did we ever actually discuss this one?
16:27:02 <arrfab> davdunc: welcome to our world :p
16:27:14 <davdunc> :)
16:27:43 <Eighth_Doctor> we never discussed it
16:28:55 <dcavalca> so, I think I agree with your comment there that this is likely mostly EPEL work
16:29:12 <dcavalca> is there anything we need specifically in Hyperscale beyond that?
16:29:45 <Eighth_Doctor> I think basically we'll have to ship libvirt with the features made available again if we can't get RHEL to ship it
16:30:11 <Eighth_Doctor> I suspect it'll be fine since that'll allow libvirt to control EL8 machines
16:30:14 <dcavalca> oh I hadn't realized this also tied into libvirt
16:30:21 <dcavalca> yeah, that makes sense
16:30:30 <Eighth_Doctor> but if they don't want it in EL9 libvirt, we're going to need to maintain our own build
16:30:44 <dcavalca> sounds like a plan
16:31:18 <Eighth_Doctor> I think so :)
16:31:28 <aekoroglu> I would like to help for libvirt builds
16:32:27 <dcavalca> so I think the next step here is to file a BZ against el9 to see if they're open to enabling this
16:32:33 <Eighth_Doctor> yeah
16:33:06 <dcavalca> aekoroglu: thanks, help is always welcome; for el8, we currently have a backport in CBS that Eighth_Doctor to update the libvirt stack
16:33:28 <dcavalca> I don't think we need/want to carry one for el9 unless we end up getting pushback from RHEL on this spice stuff
16:33:34 <Eighth_Doctor> yeah
16:34:00 <dcavalca> this might change down the road if libvirt upstream moves forward and we want to update el9 to a new version
16:34:17 <Eighth_Doctor> yup and if that happens I am prepared to maintain a libvirt backport for el9
16:34:31 <Eighth_Doctor> but I suspect we won't need to since libvirt is being continuously updated in RHEL base now
16:34:34 <dcavalca> related, there's also a long-standing ticket to get a modern qemu in EPEL: https://pagure.io/centos-sig-hyperscale/sig/issue/67
16:34:46 <dcavalca> bkircher: was working on that but I'm sure he wouldn't mind help
16:35:53 <dcavalca> jvreeland: Eighth_Doctor: any kernel updates before we move on?
16:36:43 <Eighth_Doctor> I've done a few rebases for c9s
16:36:50 <jvreeland> Later this week I'm planning on updating the work Eighth_Doctor did for c9 rebases with my c8 patches.
16:37:29 <Eighth_Doctor> My next kernel rebase is pending javierm getting me an updated simpledrm MR: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/297
16:39:27 <dcavalca> thanks folks
16:39:43 <dcavalca> alright, I think that's all for tickets
16:39:53 <dcavalca> #topic Membership
16:40:08 <dcavalca> don't think we have anything here this week?
16:40:30 <Eighth_Doctor> not that I know of
16:41:04 <dcavalca> alright
16:41:05 <dcavalca> #topic Misc
16:41:12 <dcavalca> kcwells71: you have the floor
16:41:15 <kcwells71> Cool
16:41:19 <aekoroglu> :)
16:41:27 <kcwells71> One of the things Intel would like to contribute is making current (Ice Lake) and eventually next-gen hardware available to the SIG for building and testing packages.
16:41:43 <kcwells71> Who's the best person to talk to to learn more how the current build hardware is configured and how to plug in additional new hardware?
16:41:50 <Eighth_Doctor> that's arrfab
16:41:54 <kcwells71> What are the differences, if any, between the SIG and CentOS Stream proper build hardware? How is hardware connected in? Is it different for c8s vs c9s (GitLab runners vs something else)? Those kinds of things.
16:41:58 <kcwells71> Not trying to get answers here -- ideally would set up a brief meeting with someone for some folks on my team.
16:42:10 <dcavalca> yeah, you want to talk to the CPE team
16:42:15 <Eighth_Doctor> SIGs do not have access to RHEL core infra
16:42:24 <Eighth_Doctor> that includes the infra that builds CentOS Stream itself
16:42:46 <dcavalca> the short version is that SIGs build on dedicated infra (CBS) that is walled off
16:43:16 <Eighth_Doctor> the SIG infra is completely separate and can have separate hardware additions
16:43:31 <dcavalca> the hardware itself I think is all physically in some Red Hat datacenter
16:43:32 <Eighth_Doctor> in our case, I know that CPE is prepared to set up dedicated builders for us running CentOS Hyperscale for our image build stuff, since there are issues mixing between plain and Hyperscale on host and guest for build chroots
16:43:50 <Eighth_Doctor> yeah, in Virginia if I remember correctly
16:43:54 <Eighth_Doctor> IAD2 is the DC name
16:44:05 <aekoroglu> is there any requirements for hardware's physical location ?
16:44:23 <aekoroglu> can we host it for example?
16:44:59 <dcavalca> aekoroglu: I had asked that a few years ago and it wasn't practical, but things might be different now
16:45:29 <dcavalca> recommend following up with the CPE folks (arrfab, bstinson, etc)
16:45:48 <Eighth_Doctor> file a ticket on https://pagure.io/centos-infra and follow up with arrfab
16:46:01 <dcavalca> iirc they have an open IRC meeting as well on calendar
16:46:08 <kcwells71> Nice
16:46:16 <aekoroglu> perfecto
16:47:01 <bstinson> that'd be a good discussion. feel free to cc me on that ticket too and i can get that scheduled
16:47:23 <kcwells71> bstinson: sounds good
16:47:27 <aekoroglu> +1
16:47:36 <dcavalca> on a related note: one thing I'm working on my end is putting together a diverse set of cloud instances to aid with test and devel work
16:47:56 <dcavalca> especially for things like ppc64le and s390x that are annoying to work with locally
16:48:27 <dcavalca> once I have that sorted out I'd be happy to get y'all access if it's helpful
16:48:43 <daandemeyer> Ideally we could use some of that to set up the systemd integration test suite for running on hyperscale as well
16:48:48 <Eighth_Doctor> that would be fantastic
16:48:57 <dcavalca> yeah, CI integration would be the next step
16:49:05 <Eighth_Doctor> it would make some of my work considerably less painful
16:49:10 <daandemeyer> But AFAIK, the existing centos CI uses jenkins and ubuntu uses something else entirely, seems like a pain to maintain
16:49:14 <Eighth_Doctor> speaking of which, I'm hoping to finally start working on the image build descriptions for Hyperscale 9 this week
16:49:20 <dcavalca> full disclosure: this will likely take a few months, still trying to wrangle all the approvals and stuff
16:49:31 <Eighth_Doctor> I'm going to give a go at using kiwi from the very beginning for this
16:50:13 <dcavalca> sweet, looking forward to see how that goes
16:51:05 <Eighth_Doctor> I'll probably do some live streaming later on in the week of me working on it too :)
16:51:17 <Eighth_Doctor> I just need to get the rpm+dnf defaults changed for hyperscale and I'll be set
16:51:33 <Eighth_Doctor> (move rpmdb, set sticky vendors on, etc.)
16:52:07 <bstinson> are we in open floor? because i have one quick topic to register before you folks close out
16:52:19 <dcavalca> we are
16:52:23 <dcavalca> go ahead bstinson
16:52:52 <bstinson> so, we need to talk about SIG keys for Stream 9
16:53:00 <chantra__> :) yes
16:53:12 <Eighth_Doctor> uh oh
16:53:21 <bstinson> tl;dr openssl is deprecating the sha1 algo and all of our keys have a sha1 subsignature
16:53:40 <chantra__> bstinson, you are refering to https://bugzilla.redhat.com/show_bug.cgi?id=2059424
16:53:48 <chantra__> https://bugzilla.redhat.com/show_bug.cgi?id=2059424#c7 specifically
16:54:09 <bstinson> yes, that one
16:54:12 <chantra__> >  all of our keys have a sha1 subsignature
16:54:25 <Eighth_Doctor> yup, this :(
16:54:29 <chantra__> so, from the dump I checked, the Extra key did, but so did centos proper
16:54:30 <bstinson> all of our SIG keys, that is
16:55:09 <Eighth_Doctor> yeah
16:55:22 <chantra__> https://pastebin.com/RMGc1cdw but I may be misreading the output
16:55:22 <centbot> chantra__: Paste has been copied to https://paste.centos.org/view/a7c8fbfc
16:55:22 <Eighth_Doctor> it affects everything, though some of the behavior on the keys is a little bizarre in my tests
16:55:29 <dcavalca> oh, so we need to rotate all the keys?
16:55:43 <chantra__> https://pastebin.com/tbP2nPQb is the SIG extra, I did not check any other
16:55:43 <centbot> chantra__: Paste has been copied to https://paste.centos.org/view/f11c2561
16:56:00 <bstinson> no, one sec
16:56:04 <chantra__> but I may be totally misreading pgpdump output
16:57:14 <chantra__> or there is probably 2 issues. 1) openssl is *going to* deprecate sha1, so it currently works. 2) the SIG extra key as the peculiarity "uses a SHA1 signature of a subkey that can also be used for signing. It's the equivalent of https://bugzilla.redhat.com/show_bug.cgi?id=2058497"
16:57:45 <bstinson> the distro keys are valid with the newer openssl, the SIG keys are the ones that exhibit some bad behavior
16:58:07 <aekoroglu> You are not authorized to access bug #2058497
16:58:10 <Eighth_Doctor> alas I can't read that BZ :/
16:58:14 <aekoroglu> :)
16:58:30 <chantra__> yeah found that out :)
16:59:07 <bstinson> as far as bugzillas go, we should keep to #2059424
16:59:31 <bstinson> 2058497 tracks some Red Hat keys which are affected as well
16:59:53 <michel> So not a "cipher too new" but "cipher too old" issue
17:00:08 <Eighth_Doctor> yikes
17:00:12 <bstinson> back to what we need to do though, we're planning on re-signing those subkeys and publishing those tomorrow
17:00:32 <Eighth_Doctor> does that mean we need to update distribution-gpg-keys and other things?
17:00:33 <bstinson> that requires manual intervention on user systems that have already imported the key
17:01:32 <bstinson> once they're published, we need SIGs to cut a new release of their centos-release-* packages including the re-signed pubkeys
17:01:47 <dcavalca> bstinson: to be clear, this impacts just the release packages, right?
17:01:51 <Eighth_Doctor> that I can do
17:01:59 <bstinson> (this is all going in an email to centos-devel, but i wanted to reach out directly to the affected SIGs)
17:02:06 <bstinson> dcavalca: yes, just the release packages
17:02:12 <dcavalca> thanks, definitely appreciate the heads up
17:02:13 <Eighth_Doctor> well, it means we have to resign all the published stuff too?
17:02:21 <chantra__> >  that requires manual intervention on user systems that have already imported the key
17:02:23 <bstinson> since it's the same key, we do not need to re-sign any of the RPMs that are already published
17:02:40 <chantra__> probably a stupid question... can't a new key be regenrated and imported instead?
17:02:42 <Eighth_Doctor> oh good
17:03:00 <bstinson> chantra__: if that were the case we would have to re-sign all of the rpm content
17:03:01 <chantra__> and using this new key to resign the packages
17:03:08 <Eighth_Doctor> and that would suck a lot
17:03:10 <chantra__> bstinson, ah ok
17:03:27 <dcavalca> yeah resigning all the packages would be very painful
17:03:38 <bstinson> i don't want to minimize the impact, but having folks reimport a gpg key is the cleanest
17:03:47 <dcavalca> yeah, I tend to agree
17:03:52 <bstinson> new installations will just pick up the right thing
17:04:02 <dcavalca> really glad this came up now, and not in a few months :)
17:04:06 <Eighth_Doctor> this also probably warrants a blog post so that vendors and everyone knows this kind of adjustment will be required for EL9
17:04:26 <Eighth_Doctor> and I'm glad we caught this before I made the first el9 spin :)
17:04:29 <dcavalca> bstinson: yeah, I'd recommend y'all do some wide comms around this
17:04:37 <chantra__> bstinson,  thanks for the quick turnover and heads up
17:04:45 <dcavalca> both on the specific issue, and in general on the "how to make keys that don't suck"
17:04:48 <bstinson> yes that communication is happening in concert with some RHEL communication so that we can point at each other
17:05:02 <bstinson> let me tell you about some more interesting ways to break the rpmdb too... :D
17:05:09 <bstinson> just kidding, we don't have time for that
17:05:32 <Eighth_Doctor> oh god
17:05:55 <chantra__> well, I am glad I dug into getting hsx mock to work, the day does not feel as wasted anymore :)
17:05:57 <bstinson> tl;dr watch for the heads up on centos-devel and be ready to respin your centos-release packages
17:06:05 <dcavalca> speaking of, we are over time
17:06:14 <bstinson> thanks for letting me drop in :)
17:06:21 <dcavalca> thanks everyone!
17:06:26 <dcavalca> #endmeeting