17:00:22 <tdawson> #startmeeting CentOS PaaS SIG
17:00:22 <centbot> Meeting started Wed Apr  4 17:00:22 2018 UTC.  The chair is tdawson. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:22 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:27 <tdawson> #topic roll call
17:00:41 <ogunheper> hello there
17:00:45 <rmartinelli> o/
17:00:49 <DanyC> hi all
17:00:49 <arod20832> greetings
17:01:12 <tdawson> hi ogunheper
17:01:14 <tdawson> Hi rmartinelli
17:01:17 <tdawson> Hi DanyC
17:01:18 <sdodson> tdawson: hello
17:01:21 <rmartinelli> tdawson: hi
17:01:23 <tdawson> Hi arod20832
17:01:26 <brig-man> hi
17:01:27 <tdawson> Hi sdodson
17:01:36 <tdawson> Hi brig-man
17:01:51 <ogunheper> hello tdawson
17:02:03 <tdawson> arod20832: I don't recognize you, are you usually someone else, or are you just visiting for the meeting?
17:02:34 <tdawson> arod20832: Not that we don't want more people at the meeting, just curious.
17:02:41 <arod20832> Just visiting.  Interested in learning more
17:02:54 <tdawson> arod20832: Sounds good ... welcome.
17:03:00 <arod20832> and seeing if there's some way I can contribute
17:05:08 <tdawson> #chair tdawson brig-man DanyC  sdodson ogunheper rmartinelli
17:05:08 <centbot> Current chairs: DanyC brig-man ogunheper rmartinelli sdodson tdawson
17:05:14 <tdawson> #topic OpenShift on CentOS Current Status
17:05:27 <tdawson> So, starting out with where we are
17:05:41 <tdawson> origin 3.9.0 was released this week, and we tried to build it.
17:06:17 <tdawson> turns out we have some issues with our automated script.  It isn't changing to the correct git tag, and thus, trying to build the wrong stuff.
17:07:05 <tdawson> I sent out an email to the new committee members with the problems and tests so people could see what I was seeing.
17:08:10 <tdawson> We got some good feedback from slaterx and DanyC
17:08:39 <tdawson> DanyC created  pull request ... and I believe we are currently testing to see if it works.
17:09:43 <rmartinelli> Good job guys!
17:10:15 <tdawson> I want to say thanks to everyone who jumped in to help ... we still have a bit of debugging to go with, but hopefully we are able to get this tested and on our way.
17:10:18 <ogunheper> DanyC, very well done, great...
17:10:36 <DanyC> tdawson: the PR i've done didn't tackle that issue , i went more into starting to tidy up and get rid of the deprecation warning (in next PR). Now that i am slowly getting access to CI and getting familiar with the whole infra we will just have to make the changes as slaterx said and see how it goes.
17:11:11 <bstinson> tdawson: i have a couple of things for automation
17:11:11 <tdawson> DanyC: Ah, ok.  I did notice that what he said needed to be removed was still there.
17:11:35 <tdawson> bstinson: Sure thing, nows as good a time as any
17:11:37 <DanyC> is just around testing now tbh. My key point is to be able to get a job running from my fork before we create a PR
17:11:45 <bstinson> 1.) I regenerated the cbs certificate for the paas user (it was due to expire very shortly)
17:12:07 <bstinson> 2.) I'd like to hand that off to someone active in the SIG, such that they get the emails when the cert is due to expire
17:12:42 <ogunheper> bstinson; I can get the 2nd
17:12:46 <bstinson> background: Bot accounts are just 'regular' accounts in accounts.centos.org, they follow the same 6-month regeneration cycle
17:12:56 <DanyC> bstinson: here is my thought:
17:13:00 <bstinson> we typically regenerate the cert directly in the workspace on CI
17:13:07 <bstinson> so it gets replaced properly
17:13:24 <DanyC> 1) i believe there is a wiki mentioning the steps to maintain it?
17:13:48 <tdawson> Does everyone here know who bstinson is?  He's on of the main CentOS fellows.  One of his jobs is keeping the infrastucture running smoothly.
17:14:02 <DanyC> 2) q for you and tdawson is any way a bot mailer group can be created such that the chairs of this group can be part of  ?
17:14:02 <tdawson> He does a great job at it too.
17:14:31 <bstinson> DanyC: yes! https://wiki.centos.org/SIGGuide#head-1b54333ef71a479861d61e7736f2251412c1a42c <- that's an ugly direct link in the SIGGuide
17:15:01 <DanyC> tdawson: i thought so since he is the one who should approve my bug tracker :) https://bugs.centos.org/view.php?id=14648
17:15:41 <DanyC> bstinson: great so the q i have for you is: the user is "tuser" or is whatever user i or anyone else login to ?
17:16:46 <DanyC> bstinson: and let me know on 2) what you thing - ie is there any group mailer which we can use for paas group ? rather than one of us being in charge of, i think is best all of us to be aware of it
17:16:47 <bstinson> DanyC: the name of the bot account is 'paas', and the password is the same as the credentials in the workspace. you'll get access to those (and an email with instructions on how to log into the workspace) once i handle #14648
17:17:33 <ogunheper> bstinson: should we (the others) also need to file a bug?
17:18:00 <DanyC> ogunheper: i'd say so, i'm not special here ;)
17:18:31 <ogunheper> DanyC: :)
17:19:05 <tdawson> I think you're all special :) ... but then again, if everyone is special, does that make everyone the same. :(
17:19:09 <bstinson> DanyC: the SIG can request an email alias from us if you wish. i'd say we choose a member to hand this to a SIG member for now, that way the SIG can manage its own email address for the bot account as appropriate
17:19:37 <bstinson> and yes! if you'd like access to the automation workspace in ci.c.o, please fill out a bug
17:19:42 <bstinson> https://wiki.centos.org/SIGGuide#head-593a859807e26a8299efeebda9e0aaf0d0b3a821 <- the template
17:20:16 <rmartinelli> Back to the topic: I didn't understand if the build issue is solved or not.
17:20:32 <DanyC> rmartinelli: no is not as of now, it will be shortly
17:20:50 <rmartinelli> Alright DanyC, thanks.
17:21:21 <DanyC> bstinson: cheers, we'll use a member then, ogunheper you okay to be the chosen one since you were the 1st who volunteered?
17:22:14 <ogunheper> DanyC: sure, pleasure
17:22:31 <tdawson> Following up on that, we're still working on getting a solution in place, but it's progressing nicely.
17:22:49 <tdawson> Just because DanyC was the first one with the pull request doesn't mean others can look at things and pitch in too.
17:23:07 <bstinson> ogunheper: perfect, i'll include a reminder note about this in your CI "welcome" email
17:23:17 <DanyC> tdawson: indeed
17:23:40 <ogunheper> bstinson: thanks, I'll be filing the bug after the meeting
17:24:39 <tdawson> If things take too long (which I don't think they will) we might have to do a build manually.  Although that's always a last resort, I was wondering if anyone wanted to know how to create a build manually?
17:25:09 <brig-man> tdawson:  Did we cover that last week?
17:25:36 <tdawson> No, I haven't covered building a package manually yet ... and don't plan on it this week either, I was more wondering for next week.
17:26:09 <rmartinelli> I'd like to know more about manual builds
17:26:21 <tdawson> The steps are sorta the same as the automation, though I started trying a few things differently.
17:27:12 <tdawson> OK, I'll get that planned for next week.including a write-up in the wiki.
17:27:42 <tdawson> I think it might help others understand the steps that the automation is doing.
17:28:05 <tdawson> Plus, like I said, sometimes things break.
17:28:45 <rmartinelli> Better than say: "sometimes things just work" :)
17:28:56 <tdawson> :)
17:29:04 <tdawson> It's always good when things jsut work.
17:29:41 <tdawson> The good thing about our automation is that the openshift-ansible pieces still all build correctly, including the correct git checkout.
17:30:19 <tdawson> sdodson: I've gotten one or two 3.8.0 requests ... are we ok just building whatever is latest for 3.8 ?
17:30:33 <sdodson> tdawson: Yes, that's fine.
17:31:00 <sdodson> The only updates to 3.8 will be those necessary along the upgrade path from 3.7 to 3.9 so i'd consider any changes to that branch to be critical.
17:31:01 <DanyC> tdawson: i thought 3.8 was skipped and from 3.7 people are going up to 3.9, have i misunderstood that @sdodson ?
17:31:42 <DanyC> and if my understanding is correct then who and what is the benefit of releasing 3.8 origin rpms ?
17:32:02 <brig-man> I thought that too as that is how I understood the announcement in Austin.
17:32:13 <tdawson> I believe it is for updating, that some things won't jump from 3.7 to 3.9
17:32:23 <ogunheper> DanyC: 3.8 would be in the upgrade path, but only in the upgrade path to 3.9
17:32:58 <ogunheper> It will not be allowed to be installed directly from Ansible playbooks
17:33:29 <DanyC> ogunheper: okay then. I thought by watching on the openshift-ansible repo that from 3.7 it will skip - ie no needing 3.8 rpms and jump to 3.9.
17:34:00 <ogunheper> https://docs.openshift.com/container-platform/3.9/upgrading/index.html#upgrading-introduction
17:34:12 <ogunheper> see the notification part in this section
17:34:55 <DanyC> cause up until now only the "ladder" upgrade was allowed and we had to go to each of them (and that is what i had to do from 1.4 all the way up to 3.7 internally). Okay thanks for the link ogunheper
17:35:08 <tdawson> ogunheper: Ahh ... ok, well it looks like we need to build 3.8, even if nobody is going to run it.
17:35:52 <ogunheper> tdawson: yes, that would be used, but only for a blink of an eye during upgrade :)
17:36:04 <ogunheper> for that reason
17:36:29 <ogunheper> we should not only test the installation of 3.9 rpms from scratch
17:36:43 <ogunheper> but also an upgrade path from 3.7 to 3.9
17:36:54 <ogunheper> before releasing the rpms
17:37:18 <kbsingh> that does mean we need a good way to bring up a 'populated' 3.7 cluster on demand
17:37:20 <tdawson> Yup, sounds like a good plan.
17:37:27 <kbsingh> so we can then do the update test and tear back as needed
17:38:01 <ogunheper> kbsingh: that would be great, not only for this case but in general usage also...
17:38:11 <kkeithley> Is there a repo for the src.rpms like the one at http://debuginfo.centos.org/centos/7/storage/x86_64/  ?
17:38:12 <ogunheper> do you think of a way for this?
17:38:24 <kkeithley> I tried the obvious ones
17:39:26 <avij> kkeithley: http://vault.centos.org/7.4.1708/ perhaps?
17:39:32 <DanyC> tdawson: so do we have the infra available to run the tests for upgrade/ green installation etc ?
17:39:47 <ogunheper> that topic (cluster on demand) is more related to the qa and test processes of generated packages, which I think we should elaborate on more so that we can provide a better overall quality for rpms
17:39:51 <tdawson> kkeithley: If you just need one or two source, I tend to get them from cbs  - https://cbs.centos.org/koji/
17:40:03 <tdawson> DanyC: Good question, and one that I don't know.
17:40:18 <brig-man> kkeithley: This one is included in the yum file: http://vault.centos.org/centos/7/paas/Source/openshift-origin/
17:40:29 <tdawson> DanyC: That was always something on Ari's list (and herlo too), but never got around to it.
17:40:49 <kbsingh> we can use the ci.centos.org infra, if that is suiteable ( we get bare metal )
17:40:55 <ogunheper> I think, the best place for such kind of an infra is cloud (aws, gce, or sth. like that)
17:41:04 <kkeithley> yes, vault.centos.org is what I was looking for. My browser history reminds me that I already knew about that.
17:41:05 <tdawson> I have a local Virtual Machine that I've used for installations and testing updates, but that's totally manual.
17:41:19 <kkeithley> avij++, tdawson++, brig-man++
17:41:21 <kkeithley> thanks
17:41:41 <ogunheper> we can use cloud apis to provision new vms on demand
17:42:10 <ogunheper> we can also test cloud integrations in the next steps
17:42:19 <DanyC> right so let's take few AIs cause the comms kind of mixed up a bit today and we are loosing track. We shoudl maybe then discuss over email or etherpad if there is one available
17:42:48 <DanyC> 1) tests plan for upgrades and what infra we have
17:42:48 <ogunheper> DanyC: sure, sorry
17:42:56 <tdawson> ogunheper: Ya, but that alway costs money ... not that some infrastructure of CentOS isn't funded, but cloud money is harder to get than some VM's on CentOS's infrasturcture.
17:43:29 <DanyC> 2) test plan for green deployment which we kind of have today with the infra in place
17:43:59 <ogunheper> tdawson: totally understandable, was just trying my chances :)
17:44:23 <DanyC> so for 1) while cloud is all nice and lovely i doubt we have the $$ unless someone says otherwise. Now if we stick with the infra we have i believe we need
17:44:40 <DanyC> 1.1) new CI job and maybe new slaves ?
17:45:24 <ogunheper> you mean for testing?
17:45:28 <DanyC> 1.2) the automation should be extended to cope with that . Note we should not reinvent the wheel since the openshift-ansible does have all the upgrade code
17:45:32 <DanyC> yes ogunheper
17:46:56 <DanyC> so if everyone agrees, let's iron the tasks in the github under the paas-ci git repo as issue and then we can split and work on each of them
17:47:13 <DanyC> unless tdawson has  better idea ?
17:47:28 <tdawson> That sounds as good a place to get the tasks organized as any
17:47:33 <kbsingh> +1
17:47:49 <ogunheper> +1
17:47:59 <ogunheper> just before closing the topic
17:48:23 <ogunheper> kbsingh: do you think we can find the funds for the cloud
17:48:45 <kbsingh> ogunheper: nope
17:48:52 <ogunheper> :)
17:49:07 <kbsingh> we'll need to canvas for a donor and setup a process.. its always worth a try, but its quite painful.
17:49:24 <kbsingh> specially with the cloud setup, we actually will need the providers to give us gratis accounts
17:49:57 <ogunheper> ok
17:50:06 <kbsingh> can be done, and I am happy to help with the process, but we should'nt rely on that really
17:50:33 <tdawson> So, I do want to move on to one last agenda item.
17:50:36 <DanyC> tdawson: kbsingh i believe we won't break any rules if we use https://etherpad.openstack.org/ to sketch out the plan before put it in github
17:50:45 <kbsingh> we have openstack infra in ci.centos.org - and we could get a tenant id from there.
17:51:19 <tdawson> #topic New time for weekly meetings
17:52:27 <tdawson> It's looking like the new committee is spread across timezones from the UK to NZ ... so here is the question.  How well does this meeting time work for people?
17:52:32 <bstinson> kbsingh, DanyC, ogunheper: one last point, we should really not do direct provisioning in CI. let's talk VM requirements and figure out how to get that through our existing provisioning tools
17:53:38 <rmartinelli> I'm ok with the current meeting time (it's 2pm in Brazil), but if anyone have any problems we can find another time
17:54:06 <tdawson> bstinson: Thanks for bringing that up.  I know that herlo, you, and others worked on getting that setup to work with jenkins.  we don't want to have to redo all that work again.
17:54:08 <brig-man> I'm good with the current meeting time.
17:54:08 <ogunheper> bstinson: I think we should argue all these in a separate thread and prepare a proposal for the whole pipeline that we all agree on
17:54:21 <DanyC> bstinson: fair point. should we fire an email on centos-devel mailer ?
17:54:34 <bstinson> DanyC: ci-users would be a good place
17:54:45 <kbsingh> bstinson: so thsi would work in the same way as 'request a chunk of machines' and use internally
17:55:05 <kbsingh> bstinson: not unlike say get 4 bare metal nodes, and use 1 to reinstall the other 3 :)
17:55:33 <bstinson> DanyC: feel free to tag me with your proposal and i can add some details from this side
17:55:52 <kbsingh> tdawson: if we can move the meeting -1 hr, that would help me lots and increase my ability to attend. this is 6pm to 7pm otherwise for me, and smack in the dinner timezone
17:56:19 <DanyC> i'm on same tz like @kbsingh so would work better for me too
17:57:01 <tdawson> OK, so two votes for one hour earlier, and two votes for it's fine.  I'll make sure we get everyone's feedback.
17:57:17 <tdawson> I'll probrubly send an email to centos-devel, just to make sure others don't mind as well.
17:57:20 <DanyC> bstinson: understood, will fire an email soon and then we'll take it from there .. not trying to throw any work done, i'm sure all of us who are new and with new blood coming in still miss few bits and pieces
17:58:23 <tdawson> DanyC: bstinson: ogunheper: Are you all set on testing and testing machines (sounds like some emails will go around) ?
17:58:37 <tdawson> It's getting close to the end of our meeting time, just trying to tie up all loose ends.
17:58:37 <ogunheper> is there a chance that we could move it -2 hrs?
17:58:54 <bstinson> tdawson: i think we have a good set of action items to go with
17:59:02 <bstinson> or at least i do :)
17:59:10 <tdawson> bstinson: Sounds good.
17:59:22 <tdawson> ogunheper: I'll put that in the email as one of the proposals.
17:59:26 <DanyC> and before we close, the people present here today can they try to be in this channel even when we don't have the meeting so we can flesh out together the new AIs
17:59:56 <rmartinelli> I already set this channel to autojoin.
17:59:58 <rmartinelli> :)
18:00:15 <tdawson> I try to be on this channel all the time, but when my IRC client does weird thigns, and it switches me to tdawson_, I get kicked off cuz I'm not signed in as that.
18:00:16 <DanyC> at least bouncing ideas if needed rather than email ...either way i'm good with any channel comms
18:01:05 <tdawson> Yep, I find that IRC chats are sometimes much better than emails ... though harder to document.
18:01:07 <ogunheper> tdawson: testing is not clear in my mind yet, maybe because I don't know the inf. well, I think we can discuss all this in email
18:01:33 <tdawson> ogunheper: OK, understood
18:01:56 <tdawson> OK, I'm going to close the meeting ... thanks to everyone for being here this week and the very good discussion.
18:02:09 <tdawson> You don't have to go, but I need to close the meeting.
18:02:20 <tdawson> #endmeeting