14:00:45 <bstinson> #startmeeting CBS/Infra
14:00:45 <centbot> Meeting started Mon Jan 19 14:00:45 2015 UTC.  The chair is bstinson. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:45 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:03 <bstinson> #chair alphacc Arrfab kbsingh MerlinTHP
14:01:03 <centbot> Current chairs: Arrfab MerlinTHP alphacc bstinson kbsingh
14:01:23 <bstinson> #topic Agenda
14:01:34 <MerlinTHP> Eya
14:01:34 <bstinson> #info Topic: Status Updates
14:01:35 <MerlinTHP> +H
14:01:43 <MerlinTHP> I'm in a meeting but will try and pay attention
14:01:51 <bstinson> #info Topic: Open Floor
14:02:51 <bstinson> I think since we're missing a couple of us and/or are in other meetings, we can do some quick updates and then end early
14:03:12 <bstinson> #topic Status Updates
14:04:02 <bstinson> #info Centpkg hasn't changed much, I should be pushing the patches to build from git this week
14:04:25 <bstinson> kbsingh: any news on the lookaside?
14:04:48 <alphacc> #info sigul test are ongoing but look promising as a signing solution
14:06:16 <bstinson> alphacc: nice! Arrfab got you a dev koji instance up is that right?
14:06:41 <alphacc> #info Arrfab gave me a new instance for puppetization and dark experimentation
14:07:13 <alphacc> bstinson: yes, but sigul test have been sone locally so far.
14:08:10 <mikem> what has the sigul testing entailed?
14:08:21 <mikem> I know Fedora uses it, but I thought it was rather limited
14:08:24 <alphacc> #info scl from git are building fine from git (el6 and el7 test have been done). A bug was found, if no metadata file available.
14:08:59 <alphacc> mikem: it looks good enough for our workflow but I am waiting feedback from core.
14:09:59 <alphacc> bstinson: what's your take on metadata file availability ? should we always have one in git repo ?
14:11:12 <bstinson> is it one of the buildsys tools that has trouble with the missing files?
14:13:49 <kbsingh> bstinson: not yet, I havent had the time to get into it as yet.
14:15:55 <kbsingh> bstinson: btw, that was re: lookaside cache
14:16:14 <bstinson> alphacc: if a metadata file doesn't exist {cent,r}pkg skips the "sources" step
14:16:14 <kbsingh> w.r.t the scl stuff, thats the git repo / koji use of the git repos
14:16:36 <kbsingh> if a git repo exists without a .packagename.metadata file, the get_sources.sh does not know what to do with it and exits 2
14:17:29 <kbsingh> the smartest way to fix this would be to parse the .spec if no metadata file is found, and make sure its still going to work ( ie. SOURCES? are satisfied already )
14:17:37 <kbsingh> however, I dont think we should do that in bash..
14:18:07 <kbsingh> i know alphacc has fixed this in the koji sources scripts, and i have a potential patch for howto handle this outside  ( via get_sources.sh itself )
14:20:23 <kbsingh> alphacc: does that sum up the problem ?
14:20:44 <bstinson> #topic Should Metadata files be required in dist-git repos?
14:21:20 <kbsingh> as in, make it manditory ?
14:22:21 <bstinson> i'm leaning toward no, and that we should do the RightThing(tm) in the tools if one doesn't exist
14:23:44 <MerlinTHP> So is this just to avoid having a metadata file when we have no sources in the lookaside?
14:26:11 <kbsingh> at the mo, thats the only thing in the metadata files on git.centos.org
14:26:30 <alphacc> kbsingh: fine with me.
14:26:37 <kbsingh> also, thats only non-text sources
14:26:46 <kbsingh> text sources, just go into git
14:27:04 <MerlinTHP> Sure.  But is it a common enough situation to care about?
14:27:17 <MerlinTHP> fedora git always has a sources file, which is their equivalent
14:27:57 <kbsingh> how do they checkin a zero byte file into git ?
14:28:15 <bstinson> their tools also just noop if one doesn't exist
14:29:40 <kbsingh> i dont thikn we cater to comments in get_sources.sh - if we can have it skip lines starting with #. then the import scripts can be adapted to add a metadata file always
14:29:42 <MerlinTHP> Uh, with git add?
14:29:51 <MerlinTHP> Not sure what you mean
14:31:29 <kbsingh> MerlinTHP: i dint think git liked zero byte files much
14:32:02 <MerlinTHP> It's fine with it
14:32:24 <MerlinTHP> mkdir foo ; cd foo ; git init ; touch wah ; git add wah ; git commit works just fine for me
14:32:53 <MerlinTHP> It's not a fan of empty directories, as I recall.  You might be thinking of that.
14:33:30 <MerlinTHP> https://git.fedorahosted.org/cgit/fedora-infrastructure.git/tree/scripts/distgit/setup_git_package
14:33:43 <MerlinTHP> That's the script they use to make new packages in fedora dist-git
14:33:54 <MerlinTHP> Line 104+ is the interesting bit
14:34:02 <kbsingh> no, it has ( had ? ) issues with zero byte files as well.
14:34:06 <kbsingh> atleast on centos5
14:34:58 <MerlinTHP> Hm, ok, well it's fine on 6+ :)
14:35:56 <kbsingh> ok, but that resolves that bit - if it can host empty files, we can just have a dummy placeholder
14:36:05 <kbsingh> but it would be nice to not need that file, unless its needed
14:36:39 <MerlinTHP> My personal opinion, which ain't worth much, is that it's a small corner case I'd not bother with.  Create the metadata file at repo creation.  Done.
14:37:22 <bstinson> i'm fine with that
14:38:00 <kbsingh> we can try that - but there are lots of ways for the repos to be created - we should still have the tooling skip without err if it cant find it ( and its not needed )
14:38:15 <kbsingh> in the short term, we can make sure there is a metadata file always.
14:38:21 <bstinson> that does mean that we need to sort out which packages don't have one right now and/or have get_sources treat missing metadata files the same as if there was a 0-byte file
14:38:26 <kbsingh> mikem: does that work for you ? I can go edit my import script as well
14:38:44 <mikem> re: metadata. The rh tool that publishes to dist-git will always write a metadata file, even in the unlikely case that the srpm has no sources
14:39:00 <kbsingh> mikem: yeah, i remember this conversation from last summer :)
14:41:32 <bstinson> #agreed CentOS tools will always create a .pkgname.metadata file when a repo is created, we will modify our tools to handle existing repos that are missing metadata files
14:41:54 <bstinson> #action kbsingh will look at the import script
14:42:12 <bstinson> alphacc: have you looked at get_sources yet? if not I can take that action item
14:42:44 <alphacc> bstinson: I fixed it for koji, no urgent action needed.
14:43:03 <alphacc> I think it's just a matter to decide and then modify the tool
14:43:10 <kbsingh> I've just fixed it in both the importer scripts here at my end as well
14:43:14 <mikem> I think the reason tools looked for the metadata file was as a sanity check to make sure they were in fact working in the correct directory.
14:43:58 * Arrfab joins cbs meeting .. lately
14:45:08 <bstinson> hi Arrfab!
14:46:59 <bstinson> #action bstinson to check centos-git-common to be sure it does the right thing re: metadata files
14:47:06 <bstinson> #topic Open Floor
14:47:39 <kbsingh> how are we doing on capacity in the buildsystems, do we need more machines for koji builders ?
14:48:45 <alphacc> kbsingh: it would ne nice to have few more.
14:49:25 <alphacc> kbsingh: do we have solution for faster iops for the buildroot cache dirs ?
14:50:00 <alphacc> kbsingh: you can cut buildroot creation time by 80% with ssds
14:50:01 <Arrfab> alphacc: hmm, is that using now local disks or nfs ?
14:50:08 <alphacc> Arrfab: local
14:50:26 <Arrfab> alphacc: those are SAS disks and no, we have no ssd available in that setup
14:50:41 <alphacc> Arrfab: more builders is fine then.
14:51:22 <Arrfab> alphacc: is the buildroot creation something that can be reserved on a specific node in koji ?
14:51:57 <Arrfab> otherwise the current koji-hub/koji-web can be used a specific kojid just for that task ? (thinking loud here)
14:52:06 <alphacc> Arrfab: no each builder.
14:52:12 <kbsingh> these are the mock buildroots
14:52:34 <Arrfab> ah, yeah, and using local, so nothing shared .. /me slaps himself
14:52:53 <kbsingh> no ssd's in that infra, i dont think we have any plans for ssd's either.
14:53:02 <kbsingh> are those machines running a raid10 ?
14:53:03 <alphacc> ok no problem
14:53:16 <Arrfab> kbsingh: raid1 (two 300Gb sas disks)
14:53:22 <kbsingh> iirc, its 4 disks to a node ? or is it 2
14:53:34 <kbsingh> convert those to raid0
14:53:51 <kbsingh> there is no persistant info left on any of the koji builders is there ?
14:54:38 <Arrfab> kbsingh: we can do that for the next added builder, then reinstall the existing one
14:54:42 <kbsingh> the other option is to tmpfs the buildroots, but that would severely reduce the number of builds each host can run ( and things that need large'ish roots will struggle )
14:55:08 <alphacc> kbsingh: to be fair people get used to the time it needs, I think it's not needed short term
14:55:13 <kbsingh> ok
14:55:26 <bstinson> are we pulling load metrics from koji yet?
14:55:35 <kbsingh> we can then add a couple more builder machines in there
14:55:44 <kbsingh> bstinson: i suspect zabbix might have a few stories to tell
14:55:45 <alphacc> kbsingh: ok
14:56:51 <Arrfab> bstinson: in the last 14d, cpu usage is 99.82% idle
14:57:02 <Arrfab> on average
14:58:10 <kbsingh> averages are really not worth looking at
14:58:52 <kbsingh> one interesting stat would be how many jobs were in queue, and how many times the queue grew to be more than, say 10
14:59:09 <kbsingh> and if it stays at over 10, prolly worth flagging somewhere
14:59:16 <bstinson> that might be something i can look at for my free time(heh) down the road, getting koji job load
14:59:42 <alphacc> it's all about peak when new release hit the repos. The couls monitor number of tasks (splitting regenrepo / builds)
14:59:56 <kbsingh> if you can get a number, zabbix_sender will be all you need
15:00:05 <kbsingh> alphacc: yeah
15:00:27 <bstinson> we're coming up on time, anything else for open floor?
15:00:37 * MerlinTHP is fine
15:00:50 <kbsingh> also, it might be worth plotting the diff in time from job start to job ending, to find what takes the longest and why ( i suspect its going to be the routine metadata update in centos repos )
15:00:51 <Arrfab> kbsingh: that also means generating a x509 cert for the user gathering those stats and sending those with zabbix_sender
15:00:56 <alphacc> regenrepo task could benefit from tmpfs maybe, and we can do it on few builders only.
15:01:51 <alphacc> we could have a specific builder or two which is optimized for that
15:01:59 <alphacc> s/is/are
15:02:37 <kbsingh> sounds good, so alphacc and Arrfab going to add 2 more machines to koji builders ?
15:02:47 <kbsingh> thats a good action point to close on :)
15:03:06 <alphacc> excellent and I need to run! thanks all!
15:03:14 <bstinson> yay new builders!
15:03:32 <bstinson> thanks everyone!
15:03:35 <kbsingh> ta
15:03:35 <Arrfab> #action alphacc and Arrfab to add two new builders to koji
15:03:59 <bstinson> #endmeeting