17:00:53 <tdawson> #startmeeting CentOS PaaS SIG
17:00:53 <centbot> Meeting started Wed Apr 18 17:00:53 2018 UTC.  The chair is tdawson. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:53 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:59 <tdawson> #topic roll call
17:01:33 <DanyC_> hi
17:02:07 <tdawson> Hi DanyC_
17:02:08 <brig-man> hi
17:02:11 <tdawson> Hi brig-man
17:04:51 <kincl> hi
17:04:59 <tdawson> sdodson: Are you making it to the Paas meeting this week?
17:05:02 <tdawson> Hi kincl\
17:05:24 <tdawson> sorry, over enthusiastic finger there.
17:05:29 <tdawson> Hi kincl
17:06:29 <tdawson> #chair tdawson brig-man DanyC_
17:06:29 <centbot> Current chairs: DanyC_ brig-man tdawson
17:06:38 <tdawson> We're a little slimmer in numbers this week.
17:06:42 <tdawson> Oh well, that happens.
17:06:49 <tdawson> #topic OpenShift on CentOS Current Status
17:07:08 <tdawson> Starting with rpms
17:07:12 <sdodson> hey hey
17:07:39 <tdawson> Thanks to DanyC_'s wonderful work on the automation, we now have builds for 3.7.2, 3.8.0 and 3.9.0
17:07:45 <tdawson> Hi sdodson
17:07:52 <sdodson> DanyC_: nice work!
17:08:06 <DanyC_> cheers sdodson tdawson ;)
17:08:14 <tdawson> People have actually tested them this time, all positive, except for a few misconfigurations.
17:08:53 <tdawson> All the builds have been tagged to be released, they are just waiting for the repo's to be built, the packages signed, and pushed out.
17:09:32 <tdawson> The problem is that we're right in the middle of CentOS 7.5 being released, so getting them through the pipe is a little harder.
17:09:34 <brig-man> DanyC_: great!
17:09:38 <sdodson> tdawson: awesome!
17:10:13 <tdawson> bstinson: Do you have a rough time table on how long it will take to get the origin 3.8 and 3.9 packages out?
17:11:03 <tdawson> bstinson: This is in relation to this ticket - https://bugs.centos.org/view.php?id=14689
17:12:07 <tdawson> We'll go over this in some of todays training, but for released packages, the packages have to be signed (manually) before they go into the final repo's.
17:12:28 <tdawson> They usually only do the package signing once a day, Monday through Thursday.
17:12:59 <bstinson> tdawson: we kicked a sign run a little bit ago, hopefully this will show up soon
17:13:05 <brig-man> tdawson:  Is there docs on the signing process?
17:13:12 <tdawson> bstinson: Very cool
17:13:36 <tdawson> brig-man: They are being written as we speak (I was hoping to have them written before we speak, but that only half happened)
17:13:50 <DanyC_> tdawson: not sure if kbsingh is present, i need to get access to PaaS wiki so we can edit the pages. We need to tidy up the manual steps like we spoke 1:1
17:13:52 <tdawson> https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Targets-Tags-Repos
17:14:39 <tdawson> DanyC_: True ... but looking at things, for manual builds, each different way works.  It's just that each way has it's strengths and weaknesses.
17:15:59 <DanyC_> tdawson: true but for someone w/o your experience/ history is way confusing, not to mention if they are watching what is happening on the Origin repo.
17:16:25 <tdawson> As for rpm's and automation, I'm not sure we have much more news.
17:17:09 <tdawson> When the packages reach the mirrors (http://mirror.centos.org/centos/7/paas/x86_64/) We'll send out emails and close origin issues.
17:18:03 <DanyC_> tdawson: no more news indeed, i'm going to try few things to see if i can get rid of the tito patch i had to do to get everything going and hopefully that might allow us to move closer in the direction where Origin repo is going
17:18:30 <tdawson> DanyC_: When you talk about the Origin repo, what are you meaning?
17:19:00 <DanyC_> tdawson: https://github.com/openshift/origin/pull/19331#issuecomment-381900952
17:19:18 <tdawson> DanyC_: OK, so the github repo, just needed context
17:19:30 <rmartinelli> Hey guys, sorry about the late.
17:19:58 <tdawson> Today for the training I was going to talk about the CentOS Origin rpm repo's, so I wanted to make sure.
17:20:05 <tdawson> Hi rmartinelli
17:20:46 <tdawson> #topic Training for new committee members
17:20:50 <rmartinelli> Hi tdawson
17:21:29 <tdawson> Today I wanted to talk about getting packages into repo's, as well has how to create those repo's and build targets.
17:22:30 <tdawson> We'll be dealing with two main web pages.  Our Origin workflow - https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Origin-Release-Workflow  ... and my still being written as we speak - https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Targets-Tags-Repos
17:24:25 <DanyC_> fyi tdawson for some reason i can't get to your in progress wiki https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Targets-Tags-Repos from https://wiki.centos.org/SpecialInterestGroup/PaaS so i don't know how to stay in sync and not forget about the pages. (liek i did on how to install cbs, you shared few meetings ago)
17:24:37 <tdawson> Before any repo's get created, or even the origin packages get built, we have to have a build target on cbs, along with tags.
17:25:59 <tdawson> DanyC_: Good point ... it's been added.
17:26:55 <tdawson> To create targets, and repo's, you have to create issues in CentOS's Bugzilla - https://bugs.centos.org/
17:27:43 <tdawson> I'm not positive if everyone has an account there.  If you don't, let me know.
17:28:29 <bstinson> if i may interject shortly
17:28:34 <rmartinelli> I think I'm still waiting for the account creation
17:28:35 <tdawson> Once the targets and repo's are created, then you don't have to do anything else with the bugzilla, the issue is just for their creation.  Everything else works with tagging packages.
17:28:39 <rmartinelli> lemme check
17:28:47 <tdawson> bstinson: Sure
17:29:11 <bstinson> i do recognize most of you all, but if we could be sure to keep the SIG front page up to date with membership information that helps us process those bugs
17:29:16 <rmartinelli> yup, still in NEW status.
17:29:24 <bstinson> (i.e. we check to be sure requests come from SIG members)
17:29:38 <tdawson> bstinson: Yep, I updated them last week.
17:29:49 <bstinson> tdawson: you're awesome, thanks
17:29:50 <tdawson> bstinson: Would it help if we put our IRC nick's next to the names?
17:30:44 <bstinson> we can usually parse it out from what we have in the bug, but i never say no to more info :)
17:31:05 <tdawson> bstinson: OK, I'll see if I can get that done.
17:31:22 <rmartinelli> my IRC nick and acount name is the same, so it will be more duplicate info for you. =D
17:32:22 <tdawson> So, the three issue's we need to open in the lifetime of a Origin minor release is 1) target and tag creation  2) testing repo creation and 3) release repo creation.
17:33:45 <tdawson> For the target and tag creation, I like to do mine in groups of 3.  So if you look in my example - https://bugs.centos.org/view.php?id=14545  you'll see that I did it for origin3{11,12,13}
17:34:18 <tdawson> I don't want to go much farther than 3 because origin has a history of changing numbering schemes.
17:35:23 <tdawson> But doing those in batches makes it easier for our automation, the person creating the ticket, as well as those doing the work.
17:36:13 <tdawson> Once those tickets are done, then you are able to build on the targets.
17:37:02 <tdawson> I think I mentioned it last week, but I'll mention it again.  But packages that are tagged into -candidate, are used to build the packages.
17:38:38 <tdawson> So if origin310 needs golang 1.9.4, while origin39 need golang 1.9.2, make sure that you tag golang 1.9.2 into paas7-openshift-origin39-candidate and 1.8.4 into paas7-openshift-origin310-candidate
17:39:53 <tdawson> If you want to know what packages are tagged in, you can use cbs latest-build  ( cbs latest-build --all paas7-openshift-origin39-candidate )
17:40:43 <DanyC_> tdawson: that is not done automatically today so how do you normally tag the dependencies ? i suspect using cbs ?
17:40:56 <tdawson> Or you could also use list-tagged, this will give the same result ( cbs list-tagged --latest paas7-openshift-origin39-candidate)
17:41:12 <tdawson> DanyC_: Very good question
17:41:50 <tdawson> To tag packages, use the "cbs tag-build <tag> <NVR>" command
17:42:15 <tdawson> It's the same whether for -candidate, -testing or -release, just a different tag.
17:42:39 <DanyC_> tdawson: thx
17:42:54 <tdawson> So, for this example, if 3.9 didn't have the right golang, we would do "cbs tag-build paas7-openshift-origin39-candidate golang-1.9.2-4.el7"
17:43:48 <tdawson> I *think* we only have permissions to tag packages into our paas7 tags, but just incase, be very careful which tag you use.
17:45:14 <tdawson> Packages that get tagged into -candidate take about 10-20 minutes before they show up to be used by a build.  cbs has to create a repo for it, and it deepends on how busy it is at the time.
17:46:03 <tdawson> So if you find that you need the newer version of golang, and you tag it into your -candidate tag, feel free to grab a coffee or snack before trying your build again.
17:46:24 <tdawson> That saves headaches all around.
17:47:32 <tdawson> Now, you've got your origin built, and you need to tag it into testing.  You would use the same command, just with a different tag  "cbs tag-build paas7-openshift-origin39-testing origin-3.9.0-1.el7.git.0.ba7faec"
17:48:14 <tdawson> But, if you haven't already opened an issue to create the 3.9 testing repo, that tag isn't goign to do much.
17:48:27 <kincl> tdawson: I noticed that you are building for ppc64le under the -candidate tag for multiarch, whats the future there? Will that continue? upstream origin doesn't currently release for ppc64le
17:49:00 <tdawson> kincl: Yes, our multiarch builds for ppc64le and aarch64
17:49:17 <tdawson> And when I build my golang packages, I usually build them there.
17:49:45 <kincl> gotcha
17:49:53 <tdawson> kincl: The committee member that was in charge of doing multi-arch has changed companies and is no longer around, but I talked to his successor, and he plans on picking it back up in a month or two.
17:50:29 <kincl> good deal
17:50:36 <kincl> thanks!
17:50:51 <tdawson> So, for creating testing repo's, here is an example - https://bugs.centos.org/view.php?id=14547
17:51:06 <tdawson> You'll notice that I created two testing repo's at a time.  That's a bit unusual, but doesn't have to be.
17:51:55 <tdawson> You have to have something tagged into -testing before you create a testing repo, but if you have several packages that you know (or think you know) will be released with origin, then you can go ahead and tag them into -testing and get those repo's created early.
17:53:17 <tdawson> You'll see there are two main lines there.  The first line is where you want the repo to be - https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/   see that's from that line
17:53:38 <tdawson> centos/7/paas/x86_64/openshift-origin39/
17:53:59 <tdawson> The other line is to make the CentOS maintiners life easier
17:54:09 <tdawson> + paas7-openshift-origin39-testing /|7/paas/x86_64/openshift-origin39/|7/paas/x86_64/openshift-origin39/|7/paas/x86_64/openshift-origin39/
17:54:50 <tdawson> Just change the origin39 to whichever you need to, like origin310
17:55:37 <tdawson> Can you have more than one tag going into a testing repository - yes.  You would have to tell them that at the top, and then I don't know what that bottom line would look like.
17:55:55 <tdawson> We're getting close on time, so I'm goign to speed up just a bit.
17:56:27 <tdawson> The script that creates the -testing repo's runs every two hours.  So I tell people it will take 20 minutes to 2 hours before a package shows up.
17:57:27 <tdawson> So, we have our package tested and everything looks good, we then tag it into reelease, again with the same command, but using -release  "cbs tag-build paas7-openshift-origin39-release origin-3.9.0-1.el7.git.0.ba7faec"
17:57:48 <tdawson> Again, if we don't have a release repo, we need to open an issue and create one.
17:57:56 <tdawson> BUT ... do not do this early
17:58:18 <tdawson> If you do it early, everyone will think you've already released your version and keep bugging you about your missing packages.
17:58:56 <tdawson> If you look at our example, it's similar to testing - https://bugs.centos.org/view.php?id=14689
17:59:18 <tdawson> You don't usually do two, this 3.8 and 3.9 is an exception.
17:59:36 <tdawson> But you have what tags go in, where it goes, and then the line that makes the CentOS maintainers happy.
18:00:15 <tdawson> The -release pushing has a different timeframe for going out.
18:00:39 <tdawson> First they have to create the repo, which takes some time (give them a few days)
18:00:52 <tdawson> But then, the packages have to be signed and synced out.
18:01:09 <tdawson> They only do the signing Monday through Thursday, and usually only once a day.
18:01:24 <DanyC_> tdawson: curious: why the need for this long line? "paas7-openshift-origin38-release/|7/paas/x86_64/openshift-origin38/|7/paas/x86_64/openshift-origin38/|7/paas/x86_64/openshift-origin38/" can't that be shorter "paas7-openshift-origin38-release/|7/paas/x86_64/openshift-origin38/" ?
18:01:27 <tdawson> So if you are doing a release, don't plan on it going out on a Friday.
18:01:56 <tdawson> DanyC_: You'll have to talk to the CentOS maintainers about that.
18:02:07 <DanyC_> tdawson: fair point, thx
18:02:36 <tdawson> DanyC_: I kept forgetting that line, and they kept telling me to put it in, so I put it in and things go through faster.
18:02:54 <bstinson> DanyC_: if you're truly interested, catch me sometime :) but it basically allows us to sort a bunch of packages in subdirectories and run createrepo once in a higher directory
18:03:14 <DanyC_> bstinson: cool, will do, thx
18:03:24 <bstinson> c.f. software collections
18:04:16 <tdawson> So, for a release, just be patient.  Don't do it on a Friday and expect it to go out that day.  I usually give myself a couple days leadtime, even after the repo is created.
18:05:14 <tdawson> But, if you have something like a security issue and the package has to go out immediately, go ahead and ping them  bstinson or kb, let them know the situation, and they will usually be accomodating.
18:05:51 <tdawson> We're over our time again ... any more questions?
18:06:19 <DanyC_> tdawson: i do ahve but can do after the meeting is over, is more about the notes
18:06:44 <tdawson> OK, I should be around for a bit after
18:07:51 <tdawson> Thanks to everyone for coming.   We'll talk to you next week, if not sooner.
18:07:59 <tdawson> #endmeeting