18:00:33 <dcavalca> #startmeeting Hyperscale SIG
18:00:33 <centbot> Meeting started Wed Feb 17 18:00:33 2021 UTC.  The chair is dcavalca. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:33 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:00:42 <dcavalca> oh sweet the bot works
18:00:58 <dcavalca> let's give folks a couple of minutes to join
18:01:02 <malmond> howdy
18:01:17 <jvreeland> good morning, always nice when the bots are working
18:01:34 <dcavalca> in the meantime, https://hackmd.io/@dcavalca/Hyzxsp5-u is what I put together as an agenda half an hour ago
18:01:35 <filbranden> good morning!
18:02:07 <smooge> hello
18:02:13 <tdmackey> hello
18:02:44 <dcavalca> #chair dcavalca jvreeland
18:02:44 <centbot> Current chairs: dcavalca jvreeland
18:03:43 <dcavalca> gonna go by the agenda, and then we can open the floor to whatever folks wanna talk about
18:04:04 <dcavalca> #topic Announcements
18:04:57 <dcavalca> so, I think infra-wise we have everything that we need setup
18:05:20 <dcavalca> thanks to carlwgeorge, our -release package got pushed to Extras, so our repo should be consumable directly
18:05:29 <dcavalca> there's only 'dwarves' in it for the moment
18:05:56 * King_InuYasha waves
18:06:00 <King_InuYasha> .hello ngompa
18:06:05 <King_InuYasha> meh
18:06:08 <dcavalca> hello
18:06:09 <King_InuYasha> I guess that's not a thing
18:06:30 <dcavalca> King_InuYasha: https://wiki.centos.org/SIGGuide#SIGGuide.2FSIGProcess.IRC_Channels has the bot commands
18:06:39 <dcavalca> so far I've mostly used it for logging
18:06:56 <dcavalca> in case you missed it: https://hackmd.io/bKERYvM3TAuaDs7j1XR1Mw is the agenda, I was going over announcements now
18:06:59 <King_InuYasha> zodbot is fancier
18:07:15 <King_InuYasha> cool, reading over the agenda
18:07:37 <King_InuYasha> got it
18:08:41 <dcavalca> oh, I should probably tag this stuff so it ends up in the minutes
18:08:55 <dcavalca> #info CentOS Dojo talk: https://www.youtube.com/watch?v=Kg7FqVGaZxs
18:09:28 <dcavalca> #info main package repository is ready and propagated to the mirrors, 'dwarves' is the only package in there for now
18:09:39 <dcavalca> #info centos-release-hyperscale has been released in Extras
18:09:57 <dcavalca> #info published some fairly crappy docs on how to work builds and stuff in the SIG repo
18:10:06 <dcavalca> this last one is probably worth actually talking about
18:10:29 <dcavalca> I pushed some chicken scratch notes to the repo with the stuff I did to build dwarves and the inital systemd test package
18:10:39 <dcavalca> it "works", but the workflow in general is far from optimal
18:11:11 <dcavalca> according to bstinson the plan is to get centpkg to work nicely here, so this may just be temporary
18:11:15 <King_InuYasha> nice
18:11:38 <King_InuYasha> it may make sense for there to be a cbspkg vs centpkg, since there are two koji instances to work with
18:11:53 <dcavalca> https://pagure.io/centos-sig-hyperscale/sig/blob/68d112c5bdb4f6e30102a7ada67eae6852cf9a9c/f/docs/onboarding.md
18:11:56 <King_InuYasha> but still, having the Swiss Army Knife of package infra tools working would be a huge win
18:11:59 <dcavalca> is what I have so far
18:12:19 <dcavalca> yeah, I almost started writing a crappy python wrapper for this, but decided against it, I'd rather wait for centpkg for now and improve it if needed
18:12:32 <dcavalca> koji-wise, so far we only have access to CBS
18:12:40 <michel_slm> .hello salimma
18:12:42 <michel_slm> sorry I'm late
18:12:50 <dcavalca> for centos-release-hyperscale I had to bribe carlwgeorge, as that one had to be built on mbox
18:13:26 <dcavalca> anyway, y'all feel free to dump docs in the SIG repo as you see fit as well
18:13:43 <dcavalca> I see this stuff as for internal consumption so it doesn't have to be particularly polished to be useful
18:13:57 <King_InuYasha> michel_slm: we don't have zodbot fanciness
18:14:04 <dcavalca> does anybody have any other announcements?
18:14:19 <King_InuYasha> not particularly
18:14:28 <jvreeland> nope
18:14:30 <King_InuYasha> I filed a ticket about libvirt, and I'll probably file a ticket about wireguard soon
18:14:34 <tdmackey> nope
18:15:09 <jvreeland> i did file a ticket for rasdaemon, hopping to follow/update the docs as i get things setup for it.
18:15:09 <dcavalca> cool, thanks
18:15:35 <King_InuYasha> I don't expect that either of my packages will be possible to build in the short term
18:15:45 <dcavalca> yeah, I'd say file tickets for stuff you're working on/plan to work on so we can keep track of stuff
18:15:51 <King_InuYasha> we need a way to build modules in CBS for libvirt, and we need a way to build signed kmods in CBS for wireguard
18:16:01 <dcavalca> King_InuYasha: ah, I was about to ask about modules
18:16:09 <dcavalca> we should followup with bstinson on that
18:16:31 <dcavalca> for signed kmods, my understanding from the Dojo discussion is that it's not a thing right now, but there may be a manual workaround of some kind
18:17:23 <dcavalca> ok, let's move on to
18:17:31 <dcavalca> #topic Tickets
18:17:42 <dcavalca> there's two tickets tagged with meeting to review
18:17:57 <dcavalca> #7 was tagged by Igor, who is not here though
18:18:07 <dcavalca> lemme ping him quickly
18:19:16 <dcavalca> #7 is the ticket tracking the kernel stuff
18:19:45 <dcavalca> I don't have any update on this from the FB side, we're still kinda trying to figure specifics out on our end
18:20:04 <dcavalca> during Dojo, someone mentioned there's a team in CentOS that's already building LTS kernels under altarch
18:20:12 <dcavalca> so we should definitely coordinate with them when the time comes
18:20:17 <King_InuYasha> that is pgreco
18:20:32 <King_InuYasha> he's been tracking kernel.org for armv7hl CentOS
18:21:25 <King_InuYasha> we should talk to him to see if he wants to be part of the hyperscale SIG kernel maint team, because that makes it less painful to have more stuff in the kernel enabled
18:21:32 <dcavalca> oh perfect, yeah we should follow up with him then and see if we can join forces
18:21:38 <dcavalca> King_InuYasha: wanna take point on that?
18:21:38 <King_InuYasha> since he's already having to maintain that for altarch
18:21:41 <King_InuYasha> sure
18:21:56 <dcavalca> #action King_InuYasha followup with pgreco on the altarch LTS kernels to join forces
18:22:10 <King_InuYasha> related to this, there is an open question on what to do with btrfs-progs
18:22:15 <dcavalca> jvreeland: any update on kernel stuff from your side?
18:23:21 <dcavalca> King_InuYasha: I'd say let's build it for EPEL once we have a usable kernel with btrfs in centos?
18:23:47 <King_InuYasha> does it make sense when rhel and such can't use it?
18:23:49 <jvreeland> We have a lot of interest in doing it, I've seen something on the mailinglists about a kernel SIG updates. I think our main issue going forward is still not being sure about how to organize it and the infrastructure needed to actualy release like Secureboot and moduel signing
18:24:03 <michel_slm> it seems to make sense to backport Fedora's btrfs-progs to hyperscale repo
18:24:22 <dcavalca> yeah, I'm fine with also doing btrfs-progs within Hyperscale if that's the preference
18:24:27 <dcavalca> King_InuYasha: you're the maintainer, your call :p
18:24:36 <King_InuYasha> hah
18:24:48 <King_InuYasha> I'll make a decision once we have a workflow for working with hyperscale git repos
18:24:55 <King_InuYasha> and have a kernel in the first place :)
18:25:03 <King_InuYasha> but I'm leaning towards having it in hyperscale too
18:25:35 <dcavalca> jvreeland: yeah, I'd say let's focus on actually doing something, and it is turns out the prospective Kernel SIG is a better place for it we can move it there
18:25:45 <dcavalca> but I wouldn't worry too much about org stuff at this stage
18:26:15 <dcavalca> wrt secure boot: according to bstinson, the way that'll probably have to work at least at the beginning will be via manual intervention
18:26:22 <dcavalca> as they can't put the signing keys on CBS
18:26:33 <King_InuYasha> that'll be fun
18:27:29 <jvreeland> ok, so then putting a linux repo on our centos-sig-hyperscale space on pagure and packaging it in git.centos.org on our branch should be a good first step?
18:27:45 <jvreeland> or many first steps i guess.
18:28:00 <dcavalca> yeah, I think a linux repo on Pagure is definitely a good start, and we can start using that for coordination
18:28:35 <jvreeland> #action jvreeland make pagure repo for linux
18:28:42 <dcavalca> for packaging, yeah, it looks like the kernel is just a regular package on git.centos.org, so I don't see why that wouldn't work
18:29:08 <dcavalca> worst case we'll find some corner case in the infra that'll need to be fixed :p
18:29:30 <jvreeland> #action jvreeland branch kernel in git.centos.org for hyperscale kernel
18:29:36 <dcavalca> #action dcavalca follow up with bstinson wrt secure boot and kernel builds
18:30:13 <dcavalca> King_InuYasha: ok, let's defer the btfs-progs decision for now
18:30:23 <dcavalca> anything else around kernel?
18:31:10 <dcavalca> ok, let's move on
18:31:21 <dcavalca> the other ticket on the docket is #13, which I'd tagged
18:31:29 <dcavalca> this is for the hyperscale cloud images
18:31:53 <dcavalca> now that we have an actual repo and a release package, I think we can start the conversation around having images with it enabled by default
18:32:18 <dcavalca> once we have stuff like systemd packaged, it'd be useful to have prebuilt images already running the hyperscale systemd since the beginning
18:32:31 <dcavalca> is anybody here familiar with the Cloud SIG and/or their tooling?
18:32:39 <dcavalca> I'd rather not reinvent the wheel if we can :)
18:34:15 <filbranden> I've looked into building images in the past, but that was years ago... If no one has context, I can perhaps look into it to see what it takes.
18:34:42 <dcavalca> works for me
18:34:54 <dcavalca> #action filbranden followup with Cloud SIG on how to build Hyperscale cloud images
18:35:02 <filbranden> Oh, actually that was Fedora cloud images... not sure whether there's many common parts of the tooling... But still, yeah I'm up for it.
18:35:03 <dcavalca> anything else on this?
18:35:32 <michel_slm> one thing to note, Cloud SIG seems a bit inactive so we should directly email the SIG members after filing issues on their Pagure
18:35:56 <michel_slm> King_InuYasha probably knows a lot about building images :)
18:36:10 <dcavalca> michel_slm: ah, good call
18:37:01 <dcavalca> anything else?
18:37:30 <dcavalca> ok, let's move on to
18:37:34 <dcavalca> #topic Membership
18:37:53 <dcavalca> I'd like to onboard anitazha on the SIG
18:38:13 <dcavalca> anitazha is a systemd committer and runs systemd releases at FB
18:38:25 <dcavalca> we plan to work together on getting 247 packaged within the SIG in the coming weeks
18:38:41 <jvreeland> sounds good to me
18:38:42 <filbranden> Welcome anitazha ! :-)
18:38:43 * anitazha waves
18:39:13 <dcavalca> #action dcavalca to onboard anitazha on the SIG
18:39:40 <dcavalca> looking at ACO, we have two pending requests
18:39:50 <King_InuYasha> sounds good
18:40:02 <dcavalca> michel_slm: wanna talk about the stuff you'd like to work on?
18:40:07 <King_InuYasha> dcavalca: did you track down that bug in systemd builds?
18:40:27 <michel_slm> dcavalca: I was just checking and looks like I'm not a member yet, technically
18:40:33 <dcavalca> King_InuYasha: nope, I was gonna see if it reproes with 247 as well
18:40:34 <dcavalca> https://pagure.io/centos-sig-hyperscale/sig/issue/13
18:40:54 <dcavalca> michel_slm: yes, you show up as pending in ACO
18:40:58 <michel_slm> ah ok
18:41:22 <dcavalca> michel_slm: happy to approve, but you should intro yourself for the record :)
18:41:32 <King_InuYasha> dcavalca, filbranden: I know how images are built in Fedora Cloud SIG, no clue about CentOS Cloud SIG
18:41:50 <King_InuYasha> if they're using similar tools, I can help with getting people familiar with them
18:41:52 <michel_slm> I'll be out for a month starting next week, but I plan to help with anything that align with our upstream Fedora work (so oomd, btrfs, etc.)
18:42:09 <dcavalca> King_InuYasha++
18:42:17 <dcavalca> thanks michel_slm
18:42:30 <michel_slm> are backports of packages that improve the desktop experience but are neutral for servers a good fit? e.g. bluetooth
18:42:32 <King_InuYasha> and we'll probably want to see if we can produce a transactional-update based cloud image based on our stuff :)
18:42:40 <dcavalca> oh that'd be cool
18:42:58 <dcavalca> michel_slm: if they're in base, and they don't cause regressions, I don't see why not
18:42:59 <King_InuYasha> my package review for transactional update for Fedora was approved and I'll be importing that in soon
18:43:20 <King_InuYasha> and I have a libdnf plugin to wire this up with microdnf and PackageKit
18:43:27 <michel_slm> oh, intro: I'm Michel, I support Facebook's desktop Linux fleet, and we're looking into deploying CentOS Stream on desktops too so there's a lot of alignment with this though some use cases necessarily differ
18:43:32 <King_InuYasha> #link https://code.opensuse.org/microos/libdnf-plugin-txnupd
18:43:57 <King_InuYasha> michel_slm: CentOS on the desktop is going to be... interesting
18:44:22 <dcavalca> #action dcavalca onboard michel_slm on the SIG
18:44:33 <King_InuYasha> if you folks are interested in KDE Plasma, you should hang out in the Fedora KDE SIG too, since we build Plasma for RHEL/CentOS in EPEL now
18:44:38 <michel_slm> King_InuYasha: I use it daily, no issue there :) well.. bluetooth doesn't work and suspend sometimes locks up
18:44:57 <dcavalca> we also have a pending request on ACO from
18:45:00 <dcavalca> pbisaacs (paul.isaacs@linaro.org)
18:45:07 <King_InuYasha> someone from Linaro?
18:45:11 <dcavalca> does anybody know him / has heard from him?
18:45:46 <King_InuYasha> never met him, but I know he works on HPC AArch64 enablement at Linaro
18:46:28 <dcavalca> King_InuYasha: wanna maybe ping him to get more information on this? happy to have him here once we understand what he wants to work on
18:46:40 * King_InuYasha shrugs
18:46:46 <King_InuYasha> I think that would be better for you to reach out to him
18:46:49 <dcavalca> right now the SIG is x86_64 only, but we could expand it to aarch64 is someone wanted to actuallty support that
18:46:52 <dcavalca> sure, I can do that
18:46:52 <King_InuYasha> you and I equally don't know who he is
18:46:54 <michel_slm> was he invited to this meeting?
18:47:11 <dcavalca> michel_slm: well, not directly, mostly because I only saw his pending invite now
18:47:19 <dcavalca> I did announce the meeting on centos-devel though
18:47:33 <michel_slm> that ties in with cloud images since AWS does have ARM VMs
18:47:40 <michel_slm> (Fedora Cloud images are built for both)
18:48:02 <jvreeland> dcavalca  we've talked to a few poeple about aarch64 in the SIG context happy to reach as well.
18:48:20 <dcavalca> jvreeland: sure, I'll leave this for you then
18:48:31 <jvreeland> Sounds good
18:48:33 <dcavalca> #action jvreeland followup with pbisaacs on SIG membership
18:49:05 <dcavalca> jvreeland: btw, if we do want to support aarch64, that'll require bumping and rebuilding all packages
18:49:25 <dcavalca> as currently the koji tags are only for x86_64
18:49:42 <dcavalca> ok, anything else membership-wise?
18:50:10 <King_InuYasha> I expect aarch64 is going to be something people want sooner rather than later
18:50:23 <King_InuYasha> but that's going to be complicated because of RHEL funkiness with aarch64
18:50:26 <ignatenkobrain> hello
18:50:35 <michel_slm> <King_InuYasha "but that's going to be complicat"> oh, 64k page size?
18:50:37 <ignatenkobrain> I'm very sorry for being late, had some urgent incident
18:50:40 <King_InuYasha> michel_slm: yes
18:50:45 <dcavalca> King_InuYasha: as long as someone actually commits to testing it, I have no objection
18:50:55 <dcavalca> ignatenkobrain: no worries
18:50:56 <King_InuYasha> sure, I'm just throwing that warning out there
18:51:00 <michel_slm> pbrobinson seems to imply RHEL 9 might switch back to 4k but I don't know if that's true or not
18:51:18 <dcavalca> let's move into
18:51:23 <dcavalca> #topic Misc
18:51:24 <michel_slm> (he implied Fedora switched when they discovered 64k took up too much RAM on small devices - 512MB? -- but it's too late for EL7 and 8)
18:52:08 <dcavalca> ignatenkobrain: https://www.centos.org/minutes/2021/February/centos-meeting2.2021-02-17-18.00.log.txt if you wanna catch up on logs
18:52:18 * ignatenkobrain reads through
18:52:40 <dcavalca> there's also ppc64le, but I don't really see anybody supporting that, unless we get someone from IBM interested in helping out :)
18:52:42 <jvreeland> If we're going to package our own kernels and focus on hyper scale i'm not sure we need to worry about 512MB of ram. Is there a reason to make sure aarch64 keeps the same page size as upstream RH
18:53:16 <King_InuYasha> dcavalca: we will get someone from IBM, I can practically guarantee it
18:53:25 <dcavalca> jvreeland: yeah, there's a bunch of "fun" implications, like currently btrfs filesystems cannot be shared between 4k and 64k systems (that's getting fixed)
18:53:43 <dcavalca> in general, I'd rather avoid deviating from CentOS Stream proper if we can
18:53:48 <jvreeland> makes sense
18:53:53 <King_InuYasha> IBM folks have been trying to make ppc64le and s390x better supported in CentOS for a while now
18:54:48 <dcavalca> eh, as I said above, if someone wants to actually do the work and fix stuff when it breaks, that's fine by me :)
18:54:53 <King_InuYasha> I'm in agreement here about not deviating from CentOS Stream too :)
18:55:04 <King_InuYasha> beyond what we really need to deviate
18:55:15 <ignatenkobrain> dcavalca: jvreeland King_InuYasha wrt kernel, can we in the beginning (here) agree on high-level 1) what should be the base 2) whether we really need secureboot in the very beginning 3) who owns which parts to move further
18:55:37 <ignatenkobrain> (I see we are going to have repo + branch but not clear what are the next steps)_
18:55:41 <King_InuYasha> 1. what do you mean by base?
18:55:47 <King_InuYasha> 2. we definitely need SB from the beginning
18:56:10 <dcavalca> we do need secure boot, but I think we can start doing the actual work on the kernel before that's sorted out
18:56:12 <King_InuYasha> 3. we don't even have any code to own yet, but it'll be started by dcavalca maintaining it and more folks coming on board
18:56:18 <dcavalca> we just won't be able to build real packages
18:56:24 <King_InuYasha> yup
18:56:33 <jvreeland> I was thinking just building upstream 5.4 from gregs tree is a fine first step
18:56:41 <dcavalca> as for which release we'll start from: with my FB hat on, I don't know yet
18:56:44 <ignatenkobrain> IMO that's one of the worst approaches :)
18:56:52 <ignatenkobrain> el8 kernel has backports from 5.10
18:57:02 <ignatenkobrain> so downgrading it to 5.4 sounds like a step backwards :)
18:57:19 <King_InuYasha> we'd probably start from 5.10 since it's the new longterm series
18:57:30 <King_InuYasha> and it's a lot easier to cherry-pick fixes from 5.11 to 5.10
18:58:08 <jvreeland> 5.10 works for me.
18:58:19 <dcavalca> yeah, the latest LTS seems like a reasonable pick to me
18:58:45 <ignatenkobrain> 1. I mean whether it will be something like LT, ML or something completely different
18:58:45 <ignatenkobrain> 2. dcavalca King_InuYasha does it mean that we just won't be able to ship it since it is unsigned?
18:58:45 <ignatenkobrain> 3. sure, just would like to have clear understanding who is moving which part forward. From GoodData side, we are planning on working on some more automated performance / regression testing (for our workloads) somewhere in Q2/Q3 2021, so it would be good to have some idea by then how stuff will look like and start playing with it sooner than later :)
18:59:04 <ignatenkobrain> (not sure if matrix sent this long message correctly, though)
18:59:11 <dcavalca> yeah, I can see it
18:59:36 <King_InuYasha> 1. LT, 2. Yes, sort of. We can get packages manually built and signed by c8s team, 3. sounds good
18:59:49 <dcavalca> for secure boot, I'll followup with bstinson; based on the Dojo conversations, I'm reasonable confident we can find a temporary workaround to unblock things, even if it involves bugging him to kickoff builds manually on the main koji
19:00:09 <King_InuYasha> dcavalca: we'll probably want to have a kernel source tree on pagure.io in our SIG namespace
19:00:26 <King_InuYasha> and then we can start working on kernel packaging forked from Fedora for that
19:00:35 <dcavalca> yup, I believe that's the plan
19:00:50 <ignatenkobrain> basically I am not proposing to start working on that stuff immediately but have somebody to 1) collect feedback (use-cases) 2) write down that (summary) 3) propose list of further actions :)
19:00:54 <ignatenkobrain> King_InuYasha: yep, works for me
19:00:58 <dcavalca> jvreeland: has an action to set that up
19:01:07 <dcavalca> ignatenkobrain: works for me
19:01:29 <dcavalca> ok, we're running out of time
19:01:40 <dcavalca> anything else folks want to say before we close?
19:01:44 <ignatenkobrain> 🕰️
19:02:17 <King_InuYasha> No, I'm good.
19:02:22 <jvreeland> nope
19:02:40 <dcavalca> alright, thanks everybody!
19:02:41 <michel_slm> nothing from me
19:02:47 <dcavalca> #endmeeting