17:00:22 <tdawson> #startmeeting CentOS PaaS SIG
17:00:22 <centbot> Meeting started Wed Mar 28 17:00:22 2018 UTC.  The chair is tdawson. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:22 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:27 <tdawson> #topic roll call
17:00:34 <DanyC> hi all
17:00:38 <kincl> hi
17:00:44 <brig-man> hello
17:01:08 <kbsingh> 'lo
17:01:40 <slaterx> I'm here
17:01:50 <sdodsonafk> i'm here
17:03:30 <tdawson> Hi everyone
17:03:43 <tdawson> It's good to see a full house this week
17:04:24 <tdawson> ogunheper has let me know that he will not be able to make it this week, he had some conflicts.
17:05:22 <tdawson> #chair tdawson brig-man DanyC kincl kbsingh slaterx sdodsonafk
17:05:22 <centbot> Current chairs: DanyC brig-man kbsingh kincl sdodsonafk slaterx tdawson
17:05:30 <tdawson> #topic OpenShift on CentOS Current Status
17:06:01 <tdawson> This section is going to be abreviated this week, but for the news.
17:06:47 <tdawson> The 7.3.2 origin package has not been built yet, but it has been released upstream so we can go ahead and do that.
17:07:26 <tdawson> Although it looks like it's been released, 3.9.0 hasn't really been.  We've had a couple of requests to build that, but other than scratch builds we're not doing that yet.
17:07:48 <DanyC> tdawson: you meant 3.7.2 origin ;)
17:08:01 <tdawson> :)  Yep
17:08:31 <tdawson> sdodson: For 3.7.2, should we just build the latest 3.7 openshift ansible?
17:08:38 <sdodson> tdawson: Yes, that's fine.
17:08:57 <tdawson> OK
17:09:12 <tdawson> I was hoping to do that during our training today.
17:09:42 <DanyC> tdawson: that was my next q ;) when you planning a walk through e2e ?
17:10:11 <tdawson> DanyC: Right after our status report ... which ... I think we're pretty much done with.
17:10:33 <DanyC> sounds good tdawson, cheers
17:11:00 <tdawson> Since we don't have the 3.7.2 built yet, then let's call the status report done, and move on with the training.
17:11:11 <tdawson> #topic Training for new committee members
17:12:06 <tdawson> First, a bit of news about accounts.  I have talked to most of you already, but I just barely got kbsingh the list of who I've talked to, and unless he's super human fast, the accounts aren't ready yet.
17:12:33 <tdawson> But I think everything we'll be doing today, except for the actual job submissions, you don't need accounts for.
17:13:08 <kbsingh> tdawson: you should have email :)
17:13:41 <tdawson> There are two main places we will be looking at today.  The first is the jenkins / automation page(s) - https://ci.centos.org/job/paas-ci-cd/
17:14:19 <tdawson> The second is CentOS's koji build system, called cbs, found here - https://cbs.centos.org/koji/packageinfo?packageID=3175
17:15:14 <tdawson> DanyC:  slaterx: brig-man: have any of you worked with a koji build system before?
17:15:31 <DanyC> tdawson: no
17:15:46 <brig-man> tdawson: Jenkins yes.  koji no.
17:16:23 <slaterx> tdawson: yes
17:16:33 <tdawson> OK, I won't go into a huge amount of detail, but there are a few things you have to know about it, especially with how our jenkins system works.
17:17:53 <DanyC> tdawson: i guess this is a start ;)  https://fedoraproject.org/wiki/Koji
17:18:09 <tdawson> The way we usually get it to build packages is by passing it a srpm to build.  There are other ways to get it to build, but that is how our jenkins system does it.  Basically something like "cbs build paas7-openshift-origin37-el7 origin-3.7.2-2.src.rpm"
17:18:26 <tdawson> DanyC: Yep, but it's easy to get overwhelmed there.
17:19:26 <tdawson> One of the most important things to know, for our automation, is that it will only build one package with the same NVR (Name Version Release)
17:20:06 <tdawson> So if I ran the above command twice, Koji (cbs) wouldn't let me build the package again.
17:20:28 <tdawson> But, if you want to test build something, you can give it the --scratch command.
17:21:05 <tdawson> Scratch builds don't get logged in the main database, so you can rebuild the same NVR over and over again and not have a conflict.
17:21:52 <tdawson> If you look at the cbs page that I pointed you to, you'll see several origin packages, but not the huge amount that you see if you look at the build on our jenkins side.
17:22:48 <tdawson> So, for now, that's pretty much all we need to know about Koji.  That is builds packages.  We pass it a srpm.  That it won't let you have the same NVR, unless you do scratch builds.
17:23:24 <tdawson> So now, let's move over to the Jenkins pages
17:24:14 <tdawson> Oh ... wait, one more thing.  Also that you can have different build targets.
17:25:33 <tdawson> So, you see in my command "cbs build paas7-openshift-origin37-el7 origin-3.7.2-2.src.rpm" ... paas7-openshift-origin37-el7 is a build target.  It is a place we have setup to specifically build the origin37 packages.  We have a different target for 3.4-3.12
17:26:13 <brig-man> Is that the target repository?
17:26:46 <tdawson> brig-man: no, the target repository is going to be paas7-openshift-origin37-candidate
17:27:19 <tdawson> Everything built in paas7-openshift-origin37-el7 will automatically get a paas7-openshift-origin37-candidate tag.
17:28:14 <tdawson> Also, you can tag things into that repository, and it will be part of the build repo.  This is how we get the correct version of ansible and/or golang intot he build, but tagging into the -candidate repository.
17:28:24 <tdawson> But, going back to jenkins.
17:28:46 <tdawson> If you look at the jenkins web page, you will see many build happening.
17:29:11 <tdawson> It's setup so that every time there is a successful pull request in the master branch, it does a scratch build.
17:29:41 <tdawson> This way we know when upstream does something that breaks our automation and/or their own packages.
17:30:03 <tdawson> If we look at one of these builds - https://ci.centos.org/job/paas-ci-cd/1360/
17:30:25 <tdawson> And then look at it's parameters - https://ci.centos.org/job/paas-ci-cd/1360/parameters/
17:30:59 <tdawson> We will see the different options that we have to work with to do a build.
17:31:45 <tdawson> When you get login permissions, you will be able to create a manual build, and these are the parameters that you use.
17:31:48 <DanyC> tdawson: so i guess today the OA/ Origin builds are automatically created with a github release ? all coming from this build ?
17:32:23 <tdawson> DanyC: Yes and No.  Only the master branch, and those are all scratch builds.
17:33:17 <tdawson> I *thought* ari had it setup so that when a new tag is created, or package released, it will do a new non-scratch build of it.  But since 3.7.2 came out, and I didn't see a build, that might not be happening.
17:33:41 <tdawson> So we will need to do a manual build.
17:34:38 <tdawson> Although the variables are meant to be obvious, and there is some descriptions, I'm going to go through them, just so we have it.
17:34:51 <slaterx> tdawson: Can we build non-scratch builds straight away or should we do a scratch first to confirm then a real one?
17:35:22 <tdawson> ORIGIN_VERSION and OA_VERSION are the git tag's from the origin and openshift-ansible git repositories.
17:35:55 <tdawson> slaterx: That is totally up to you, on how confident you are that a build will work.  There is no automation in place that forces you to do a scratch build first.
17:36:48 <tdawson> BUILD_TARGET is what we talked about above.  This is optional because it usually can figure it out.  But if you want to have it build someplace else, like the multiarch target, then you would put in in there.
17:38:01 <tdawson> SCRATCH is what we talked about above.  Koji (cbs) will do a scratch build.  This is good for testing.  But if you want to download the packages that it builds, then you have to know the task number of the package... which should be in the jenkins logs.
17:38:32 <tdawson> BE tells it to build off the master branch or not.  For our upcoming example, we will make sure this is not checked.
17:39:19 <tdawson> BUILD_ORIGIN and BUILD_OA are just like they sounds.  If you want just openshift ansible, have just it checked, just origin, have just it checked, or both have them both checked.
17:39:49 <DanyC> tdawson: all clear so far. If you don't mind asking a different question just so i get the full picture: in terms of the e2e flow: from a git repo side, i guess someone with power within RH does create the tag, which then invoke another build job to generate the Github Release and then we jump on this side of the world with CBS ?
17:40:12 <tdawson> DanyC: That is correct
17:40:53 <tdawson> DanyC: So we have no control over the upstream git repo, we are only consumers of it.
17:41:11 <slaterx> tdawson: when we tell not to build from master branch, will it build from the tag?
17:41:41 <tdawson> slaterx: correct
17:41:46 <DanyC> tdawson: crystal clear, thx
17:43:13 <tdawson> So, I'm going to do the 3.7.2 build ... to get setup I have the upstream origin and openshift_ansible checked out, done a git pull on them both, and then a "git tag" on them both.
17:44:19 <tdawson> You can't see, but if you are logged in with permissions there is a "build with parameters" tab.  If you click it, it looks just like the one we were at above,https://ci.centos.org/job/paas-ci-cd/1360/parameters/ except there is a "BUILD" button.
17:45:52 <slaterx> tdawson: when you say setup, you mean you have the tags for both origin and openshift-ansible, correct?
17:46:02 <tdawson> So now there is a job running, and you can see the parameters I gave it - https://ci.centos.org/job/paas-ci-cd/1361/parameters/
17:46:13 <tdawson> slaterx: correct
17:46:37 <tdawson> slaterx: So if you don't see any tags when you do "git tag" then you might need to do "git fetch tags"
17:47:54 <tdawson> While this is going through it's stuff, let's look at one other thing, the code behind the automation.  I don't expect you to have to know it all if you don't want to, but it's good to know where it is.
17:48:25 <tdawson> https://github.com/CentOS-PaaS-SIG/paas-sig-ci
17:48:49 <tdawson> Also, Ari setup a page that shows the workflow of the automation - https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Origin-Release-Workflow
17:50:15 <tdawson> Looking at the time, this build isn't going to finish before the meeting ends.  Are there any questions about automation and the builds, before I go into what we do after we build the packages?
17:50:45 <DanyC> tdawson: not from my side
17:51:01 <slaterx> tdawson: nope
17:51:04 <brig-man> tdawson: I'm good.
17:51:40 <tdawson> When a package it built, it is automatically tagged into -candidate
17:52:38 <tdawson> The candidate repos's are found here https://cbs.centos.org/repos/ ... and our particual one, for our example is here https://cbs.centos.org/repos/paas7-openshift-origin37-candidate/x86_64/os/
17:53:15 <tdawson> The first thing you want to do after a package builds, is do minor testing (does it install) and give it a -testing tag.
17:53:56 <tdawson> If we look at the last origin 3.7.1-2 package, we can see all the tags it has - https://cbs.centos.org/koji/buildinfo?buildID=22360
17:54:17 <DanyC> tdawson: the wiki mentioned https://cbs.centos.org/repos/paas7-openshift-future-candidate/x86_64/os/ while you mentioned https://cbs.centos.org/repos/paas7-openshift-origin37-candidate/x86_64/os/ . Any diff between the 2 ?
17:54:23 <tdawson> the tagging workflow is -candidate => -testing => release
17:54:46 <tdawson> DanyC: Ohh ... good point, and we should probrubly update the wiki
17:55:53 <tdawson> The -future- target and repo's were how we were originally testing newer origin releases.  It wasn't really working out and we went to have a target and repo for each release.
17:56:33 <tdawson> Right now, despite the name, the -future- repo and target is the least maintained.
17:56:42 <tdawson> Sorta like what happens to a Worlds Fair
17:56:47 <tdawson> But I digress
17:57:30 <tdawson> After a package is tested to your liking, then you tag it into -release, and that is when it goes live.
17:57:42 <tdawson> Oh, the tagging repo's are here. - https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/
17:58:00 <tdawson> I'm not sure why it's called "buildlogs", but that's the testing repo areas.
17:58:22 <tdawson> Well, actually, this is a better link - https://buildlogs.centos.org/centos/7/paas/x86_64/
17:58:37 <tdawson> One last thing.  When we do a release, you'll notice there are two -release tags.
17:59:14 <tdawson> paas7-openshift-origin37-release is the repo's for is someone wants to stay on just origin37, the point at that repo.
17:59:30 <tdawson> paas7-openshift-origin-release is the "latest release" repo.
17:59:45 <tdawson> So that repo always has the latest release.
17:59:52 <tdawson> Right now that is the 3.7 releases.
18:00:15 <DanyC> tdawson: a silly q but how do you currently promote/ tag the packges between the repos ?
18:00:36 <tdawson> But once 3.9.0 is released upstream, build, tested and released in CentOS, we will not put any more 3.7 packages in there.
18:01:09 <DanyC> tdawson: i get is a manual step (as per the wiki) but asking so we can then improve the wiki, unless info is already there
18:01:39 <tdawson> DanyC: Good question.  FOr the most part, you just have to use a koji/cbs command.  So for this one we do "cbs tag-pkg paas7-openshift-origin37-release origin-3.7.1-2.el7"
18:02:41 <DanyC> tdawson: cool. This cbs utility i guess we need to install it on our dev env ? if so where do we get the pkg, etc  ?
18:02:43 <tdawson> DanyC: It's something that *could* be automated.  At least automate moving it into -testing.  But for the final release I've always thought it should be done by hand, just so there is some human saying they've checked it.
18:03:00 <DanyC> tdawson: agree with that
18:03:42 <tdawson> Also a very good question ... there is a good centos cbs web page ... looking
18:03:52 <brig-man> tdawson: I'm wondering as k8s/Openshift are stabilizing, they will be releasing bug fix releases and with only doing the latest, Centos won't get the bug fix releases.
18:04:36 <tdawson> I think this is the page that has how to install and use cbs https://wiki.centos.org/SIGGuide#head-60619fe2d6d558e994eacccad20539c8a050585c
18:05:38 <tdawson> brig-man: That's only for that one repository.
18:06:17 <tdawson> brig-man: If they were to have a 3.6.5 release, we *should* be able to build that, tag it through our system, and it would go into the origin36 repos
18:06:46 <brig-man> tdawson: OK.  Got to go.  Another meeting.  I'll catch up via email or just reading the rest of this.
18:07:02 <tdawson> Oh ... I need to wind this down anyway, there is a meeting after ours.
18:07:10 <DanyC> tdawson: thx for the CBS link, now got the full picture
18:07:23 <tdawson> Thank you everyone for showing up
18:07:32 <tdawson> And especially thank you to everyone who has volunteered.
18:07:45 <tdawson> I really hope with the larger number the worload is lower and easier.
18:07:51 <slaterx> tdawson: glad to help :-)
18:08:03 <DanyC> tdawson: cheers for taking time to take us through
18:08:32 <tdawson> It's my pleasure, the more I get the information to you all, the quicker I can shift the load.
18:09:15 <tdawson> Like I said, I need to end this meeting ... talk to you all next week, but probrubly sooner.
18:09:27 <tdawson> #endmeeting