You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Andor Molnar <an...@apache.org> on 2020/08/14 10:52:56 UTC
Re: [IMPORTANT] - Migration to ci-builds.a.o
Hi devs,
There’s some movement from Apache on Jenkins migration. Github account is now available on the new instance which will be useful for PreCommit builds. I started the tests.
Deadline (15 Aug) is approaching. Is there anything else which has fallen through the cracks and has to be migrated to the new instance?
Andor
> On 2020. Jul 24., at 19:15, Patrick Hunt <ph...@apache.org> wrote:
>
> On Fri, Jul 24, 2020 at 3:05 AM Andor Molnar <an...@apache.org> wrote:
>
>> Hi,
>>
>> Jenkinsfile is now committed to master, 3.5 and 3.6 branches. Build is
>> running here:
>>
>> https://ci-hadoop.apache.org/view/ZooKeeper/job/zookeeper-master-maven-multipipeline/
>>
>> I use standard Git interface to access the repo, because using GitHub
>> means that branch discovery has to run through the github API. The problem
>> with this is that I haven’t specified credentials and anonymous access is
>> limited by some quota, meaning the discovery process takes ages (1-2 hours).
>>
>> Builds are running fine except I see a lot of concurrent builds on master
>> without any meaningful SCM change. Odd. I will change pollSCM trigger to
>> run @hourly and will see how it goes.
>>
>> What’s outstanding?
>>
>> - JDKs: currently only two: 8, 11 (LTS versions). What other JDKs would
>> you like to run against?
>>
>> - GitHub Pull Requests precommit job: I have no idea how to do this, but I
>> suspect we have to use the GitHub Api for this to work.
>>
>>
> Not sure if this is helpful but I saw it recently:
>
>> we need ability to trigger builds on pull request creation/updates which
>
>> requires the plug-in below or similar:
>
>> https://plugins.jenkins.io/ghprb/
>
>
>
> That plugin is deprecated and not recommended for use .
>
>
> I have installed the *branch source plugin instead*, hopefully that works
> for
>
> you.
>
>
> If not, let's investigate further what else we can do
>
>
>
>> - Windows build: there’s no Windows agent available currently in the new
>> instance.
>>
>> - Ant builds: do we need this?
>>
>> Please share your thoughts.
>>
>> Regards,
>> Andor
>>
>>
>>
>>
>>> On 2020. Jul 23., at 22:08, Andor Molnar <an...@apache.org> wrote:
>>>
>>> Hi folks,
>>>
>>> PR is available: https://github.com/apache/zookeeper/pull/1409
>>>
>>> Andor
>>>
>>>
>>>
>>> On Thu, 2020-07-23 at 11:32 +0200, Andor Molnar wrote:
>>>> Created a Jira for the task:
>>>>
>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3896
>>>>
>>>> Andor
>>>>
>>>>
>>>>
>>>>> On 2020. Jul 21., at 12:10, Andor Molnar <an...@apache.org> wrote:
>>>>>
>>>>> Where’s that example…?
>>>>>
>>>>>
>>>>>
>>>>>> On 2020. Jul 20., at 20:02, Enrico Olivelli <eo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Il Lun 20 Lug 2020, 19:40 Andor Molnar <an...@apache.org> ha
>>>>>> scritto:
>>>>>>
>>>>>>> Hi Enrico,
>>>>>>>
>>>>>>> No worries, I only created a few jobs to make some progress,
>>>>>>> but feel free
>>>>>>> to ignore that and do it in a better way. The “View” or
>>>>>>> “Folder” that I was
>>>>>>> adding jobs is
>>>>>>>
>>>>>>> https://ci-hadoop.apache.org/view/ZooKeeper/
>>>>>>>
>>>>>>>
>>>>>>> Andor
>>>>>>>
>>>>>>
>>>>>> This is an example from Apache Maven project. It is very complex
>>>>>> because
>>>>>> tests are in another repo and for lots if other reasons. We just
>>>>>> have to
>>>>>> create a simpler file.
>>>>>>
>>>>>> If nobody volunteers I can try to spend some time but I won't
>>>>>> have a fast
>>>>>> pace these weeks
>>>>>>
>>>>>>
>>>>>> Enrico
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> On 2020. Jul 20., at 19:34, Enrico Olivelli <
>>>>>>>> eolivelli@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Il Lun 20 Lug 2020, 19:02 Patrick Hunt <ph...@apache.org> ha
>>>>>>>> scritto:
>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2020 at 9:47 AM Enrico Olivelli <
>>>>>>>>> eolivelli@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Il Lun 20 Lug 2020, 18:41 Patrick Hunt <ph...@apache.org>
>>>>>>>>>> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> On Sat, Jul 18, 2020 at 12:20 PM Andor Molnar <
>>>>>>>>>>> andor@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> Hi Pat,
>>>>>>>>>>>>
>>>>>>>>>>>> I have admin rights in the new system too and
>>>>>>>>>>>> probably can work on
>>>>>>>>> this
>>>>>>>>>>> on
>>>>>>>>>>>> Monday.
>>>>>>>>>>>> What’s “matrix” config? Shouldn’t we just replicate
>>>>>>>>>>>> the same jobs on
>>>>>>>>>> the
>>>>>>>>>>>> new instance?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Not sure on the exact name/feature but "matrix" is
>>>>>>>>>>> basically the
>>>>>>>>> ability
>>>>>>>>>> to
>>>>>>>>>>> say "run this build/test across a set of JDK versions"
>>>>>>>>>>> rather than a
>>>>>>>>>> single
>>>>>>>>>>> version. As a result, instead of 3 jobs for zk3.6.0,
>>>>>>>>>>> jdk 1/2/3 you end
>>>>>>>>> up
>>>>>>>>>>> with a single job which runs three times, one for each
>>>>>>>>>>> jdk type and
>>>>>>>>>>> summarizes the results. I've seen this before, I assume
>>>>>>>>>>> it's a feature
>>>>>>>>> of
>>>>>>>>>>> jenkins itself?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We should use Jenkins files and this configuration will
>>>>>>>>>> be easy and
>>>>>>>>>> committed to git
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The ability to do "gitops" would be amazing!
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yes.
>>>>>>>> But I saw on Slack that Andor already started to create new
>>>>>>>> jobs.
>>>>>>>> We should do only one way in order not to waste time.
>>>>>>>>
>>>>>>>> Andor can you please share your work?
>>>>>>>> Thanks for doing it
>>>>>>>>
>>>>>>>> Enrico
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Patrick
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Enrico
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Patrick
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Andor
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020. Jul 17., at 2:51, Patrick Hunt <
>>>>>>>>>>>>> phunt@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I updated the job I linked earlier based on what's
>>>>>>>>>>>>> the latest on
>>>>>>>>> the
>>>>>>>>>>>> legacy
>>>>>>>>>>>>> jenkins. It ran successfully
>>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/215/
>>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't replicate every config setting - main gap
>>>>>>>>>>>>> is the spotbugs
>>>>>>>>>>>>> post-build, which seems to be missing from the new
>>>>>>>>>>>>> jenkins plugins.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's just maven master though. Not sure about the
>>>>>>>>>>>>> rest. Can we do
>>>>>>>>>>> more
>>>>>>>>>>>> of
>>>>>>>>>>>>> a "matrix" config in the new system vs cloning all
>>>>>>>>>>>>> the time?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:31 PM Patrick Hunt <
>>>>>>>>>>>>> phunt@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:22 PM Patrick Hunt <
>>>>>>>>>>>>>> phunt@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 12:56 PM Enrico
>>>>>>>>>>>>>>> Olivelli <
>>>>>>>>>>> eolivelli@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> FYI
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fun. I do notice it says "Hadoop and related
>>>>>>>>>>>>>>> projects have their
>>>>>>>>>> own
>>>>>>>>>>>>>>> migration path to follow" - any insight on
>>>>>>>>>>>>>>> that? We are or are
>>>>>>>>> not
>>>>>>>>>>>> lumped
>>>>>>>>>>>>>>> in? I would assume we are?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This (eventual migration) came up a while back
>>>>>>>>>>>>>>> on the Hadoop PMC
>>>>>>>>>> and
>>>>>>>>>>> I
>>>>>>>>>>>>>>> volunteered to try for us (ZK). I was never
>>>>>>>>>>>>>>> able to get it to
>>>>>>>>>> work, I
>>>>>>>>>>>>>>> provided feedback to infra but they never got
>>>>>>>>>>>>>>> back, as such we
>>>>>>>>>> have
>>>>>>>>>>> a
>>>>>>>>>>>>>>> project here that's not working with some basic
>>>>>>>>>>>>>>> dependencies
>>>>>>>>>> missing:
>>>>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, we can try again. Can we verify
>>>>>>>>>>>>>>> where ZK is supposed
>>>>>>>>> to
>>>>>>>>>>>> land?
>>>>>>>>>>>>>>> Perhaps we can try to delete and recreate the
>>>>>>>>>>>>>>> POC job I created
>>>>>>>>> at
>>>>>>>>>>> that
>>>>>>>>>>>>>>> link to see if we can get it working?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see another email thread on the list saying
>>>>>>>>>>>>>> that we are part of
>>>>>>>>>> said
>>>>>>>>>>>>>> "related projects". We are expected to move to
>>>>>>>>>>>>>> http://ci-hadoop.apache.org/ within 4 weeks.
>>>>>>>>>>>>>> Seems nodes are
>>>>>>>>>> already
>>>>>>>>>>>>>> being removed/migrated from the "H#" pool.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are over 400 plugins on the current
>>>>>>>>>>>>>> builds.apache.org -
>>>>>>>>> most
>>>>>>>>>> of
>>>>>>>>>>>> which
>>>>>>>>>>>>>> we don't need any more, or are replaced with
>>>>>>>>>>>>>> different plugins on
>>>>>>>>>> the
>>>>>>>>>>>> new
>>>>>>>>>>>>>> system. I expect there may be some plugins we
>>>>>>>>>>>>>> still need to
>>>>>>>>> install
>>>>>>>>>> to
>>>>>>>>>>>> get
>>>>>>>>>>>>>> you going again, which is why it is vitally
>>>>>>>>>>>>>> important that you
>>>>>>>>> start
>>>>>>>>>>>>>> testing and migrating your jobs over *now*. You
>>>>>>>>>>>>>> should all have
>>>>>>>>>> auth.
>>>>>>>>>>>>>> Any questions, feel free to email the
>>>>>>>>> hadoop-migrations@apache.org
>>>>>>>>>>>> list if
>>>>>>>>>>>>>> you are one of the projects listed below. The
>>>>>>>>>>>>>> rest of you, not
>>>>>>>>>>> listed, a
>>>>>>>>>>>>>> similar email to this one will be posted for you
>>>>>>>>>>>>>> shortly on
>>>>>>>>>>> builds@a.o.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> full details
>>>>>>>>>>>>>>
>>>>>>>
>> https://lists.apache.org/thread.html/r21c9d40cdbf5461143dd7eb4ff48a200c2fd20c50e946884f61318fd%40%3Cbuilds.apache.org%3E
>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>>> Da: Gavin McDonald <gm...@apache.org>
>>>>>>>>>>>>>>>> Date: Gio 16 Lug 2020, 18:33
>>>>>>>>>>>>>>>> Subject: [IMPORTANT] - Migration to ci-
>>>>>>>>>>>>>>>> builds.a.o
>>>>>>>>>>>>>>>> To: builds <bu...@apache.org>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This NOTICE is for everyone on
>>>>>>>>>>>>>>>> builds.apache.org. We are
>>>>>>>>>> migrating
>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> Cloudbees based Client Master called
>>>>>>>>> https://ci-builds.apache.org
>>>>>>>>>> .
>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> migrations of all jobs needs to be done
>>>>>>>>>>>>>>>> before the switch off
>>>>>>>>> date
>>>>>>>>>>> of
>>>>>>>>>>>>>>>> 15th
>>>>>>>>>>>>>>>> August 2020, so you have a maximum of 4
>>>>>>>>>>>>>>>> weeks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is no tool or automated way of
>>>>>>>>>>>>>>>> migrating your jobs, the
>>>>>>>>>>>>>>>> differences in the platforms, the plugins and
>>>>>>>>>>>>>>>> the setup makes it
>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>> to do in a safe way. So, you all need to
>>>>>>>>>>>>>>>> start creating new jobs
>>>>>>>>>> on
>>>>>>>>>>>>>>>> ci-infra.a.o and then , when you are happy,
>>>>>>>>>>>>>>>> turn off your old
>>>>>>>>>> builds
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> builds.a.o.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There are currently 4 agents over there ready
>>>>>>>>>>>>>>>> to take jobs,
>>>>>>>>> plus a
>>>>>>>>>>>>>>>> floating
>>>>>>>>>>>>>>>> agent which is shared amongst many masters
>>>>>>>>>>>>>>>> (more to come). I
>>>>>>>>> will
>>>>>>>>>>>> migrate
>>>>>>>>>>>>>>>> away 2 more agents from builds.a.o to ci-
>>>>>>>>>>>>>>>> builds.a.o every few
>>>>>>>>>> days,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> will keep an eye of load across both and
>>>>>>>>>>>>>>>> adjust accordingly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If needed, create a ticket on INFRA jira for
>>>>>>>>>>>>>>>> any issues that
>>>>>>>>> crop
>>>>>>>>>>> up,
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> email here on builds@a.o. there may be one or
>>>>>>>>>>>>>>>> two plugins we
>>>>>>>>> need
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> install/tweak etc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We will be not using 'Views' at the top
>>>>>>>>>>>>>>>> level, but rather we
>>>>>>>>> will
>>>>>>>>>>> take
>>>>>>>>>>>>>>>> advantage of 'Folders Plus' - each project
>>>>>>>>>>>>>>>> will get its own
>>>>>>>>> Folder
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> authorisation access to create/edit jobs etc
>>>>>>>>>>>>>>>> within that
>>>>>>>>> folder. I
>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> create these folders as you ask for them to
>>>>>>>>>>>>>>>> start with. This
>>>>>>>>> setup
>>>>>>>>>>>> allows
>>>>>>>>>>>>>>>> for credentials isolation amongst other
>>>>>>>>>>>>>>>> benefits, including but
>>>>>>>>>> not
>>>>>>>>>>>>>>>> limited
>>>>>>>>>>>>>>>> to exclusive agents (Controlled Agents) for
>>>>>>>>>>>>>>>> your own use ,
>>>>>>>>> should
>>>>>>>>>>> you
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> any project targeted donations of agents.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As with other aspects of the ASF, projects
>>>>>>>>>>>>>>>> can choose to just
>>>>>>>>>> enable
>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> committers access to their folder, just ask.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We will re-use builds.apache.org as a CNAME
>>>>>>>>>>>>>>>> to ci-builds.a.o
>>>>>>>>> but
>>>>>>>>>>> will
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> be setting up any forwarding rules or
>>>>>>>>>>>>>>>> anything like that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, please, get started *now *on this so you
>>>>>>>>>>>>>>>> can be sure we are
>>>>>>>>>> all
>>>>>>>>>>>>>>>> completed before the final cutoff date of
>>>>>>>>>>>>>>>> 15th August 2020.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any questions - I expect a few (dozen :) ) -
>>>>>>>>>>>>>>>> ask away and/or
>>>>>>>>> file
>>>>>>>>>>>> INFRA
>>>>>>>>>>>>>>>> tickets.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hadoop and related projects have their own
>>>>>>>>>>>>>>>> migration path to
>>>>>>>>>> follow,
>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> cut off date, Cassandra, Beam, CouchDB have
>>>>>>>>>>>>>>>> already migrated and
>>>>>>>>>> are
>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>> very well in their new homes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lets get going ...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Gavin McDonald*
>>>>>>>>>>>>>>>> Systems Administrator
>>>>>>>>>>>>>>>> ASF Infrastructure Team
>>>
>>
>>
Re: [IMPORTANT] - Migration to ci-builds.a.o
Posted by Andor Molnar <an...@apache.org>.
Answering myself:
Flaky Test Report
> On 2020. Aug 14., at 12:52, Andor Molnar <an...@apache.org> wrote:
>
> Hi devs,
>
> There’s some movement from Apache on Jenkins migration. Github account is now available on the new instance which will be useful for PreCommit builds. I started the tests.
>
> Deadline (15 Aug) is approaching. Is there anything else which has fallen through the cracks and has to be migrated to the new instance?
>
> Andor
>
>
>
>> On 2020. Jul 24., at 19:15, Patrick Hunt <ph...@apache.org> wrote:
>>
>> On Fri, Jul 24, 2020 at 3:05 AM Andor Molnar <an...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Jenkinsfile is now committed to master, 3.5 and 3.6 branches. Build is
>>> running here:
>>>
>>> https://ci-hadoop.apache.org/view/ZooKeeper/job/zookeeper-master-maven-multipipeline/
>>>
>>> I use standard Git interface to access the repo, because using GitHub
>>> means that branch discovery has to run through the github API. The problem
>>> with this is that I haven’t specified credentials and anonymous access is
>>> limited by some quota, meaning the discovery process takes ages (1-2 hours).
>>>
>>> Builds are running fine except I see a lot of concurrent builds on master
>>> without any meaningful SCM change. Odd. I will change pollSCM trigger to
>>> run @hourly and will see how it goes.
>>>
>>> What’s outstanding?
>>>
>>> - JDKs: currently only two: 8, 11 (LTS versions). What other JDKs would
>>> you like to run against?
>>>
>>> - GitHub Pull Requests precommit job: I have no idea how to do this, but I
>>> suspect we have to use the GitHub Api for this to work.
>>>
>>>
>> Not sure if this is helpful but I saw it recently:
>>
>>> we need ability to trigger builds on pull request creation/updates which
>>
>>> requires the plug-in below or similar:
>>
>>> https://plugins.jenkins.io/ghprb/
>>
>>
>>
>> That plugin is deprecated and not recommended for use .
>>
>>
>> I have installed the *branch source plugin instead*, hopefully that works
>> for
>>
>> you.
>>
>>
>> If not, let's investigate further what else we can do
>>
>>
>>
>>> - Windows build: there’s no Windows agent available currently in the new
>>> instance.
>>>
>>> - Ant builds: do we need this?
>>>
>>> Please share your thoughts.
>>>
>>> Regards,
>>> Andor
>>>
>>>
>>>
>>>
>>>> On 2020. Jul 23., at 22:08, Andor Molnar <an...@apache.org> wrote:
>>>>
>>>> Hi folks,
>>>>
>>>> PR is available: https://github.com/apache/zookeeper/pull/1409
>>>>
>>>> Andor
>>>>
>>>>
>>>>
>>>> On Thu, 2020-07-23 at 11:32 +0200, Andor Molnar wrote:
>>>>> Created a Jira for the task:
>>>>>
>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3896
>>>>>
>>>>> Andor
>>>>>
>>>>>
>>>>>
>>>>>> On 2020. Jul 21., at 12:10, Andor Molnar <an...@apache.org> wrote:
>>>>>>
>>>>>> Where’s that example…?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 2020. Jul 20., at 20:02, Enrico Olivelli <eo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Il Lun 20 Lug 2020, 19:40 Andor Molnar <an...@apache.org> ha
>>>>>>> scritto:
>>>>>>>
>>>>>>>> Hi Enrico,
>>>>>>>>
>>>>>>>> No worries, I only created a few jobs to make some progress,
>>>>>>>> but feel free
>>>>>>>> to ignore that and do it in a better way. The “View” or
>>>>>>>> “Folder” that I was
>>>>>>>> adding jobs is
>>>>>>>>
>>>>>>>> https://ci-hadoop.apache.org/view/ZooKeeper/
>>>>>>>>
>>>>>>>>
>>>>>>>> Andor
>>>>>>>>
>>>>>>>
>>>>>>> This is an example from Apache Maven project. It is very complex
>>>>>>> because
>>>>>>> tests are in another repo and for lots if other reasons. We just
>>>>>>> have to
>>>>>>> create a simpler file.
>>>>>>>
>>>>>>> If nobody volunteers I can try to spend some time but I won't
>>>>>>> have a fast
>>>>>>> pace these weeks
>>>>>>>
>>>>>>>
>>>>>>> Enrico
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> On 2020. Jul 20., at 19:34, Enrico Olivelli <
>>>>>>>>> eolivelli@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Il Lun 20 Lug 2020, 19:02 Patrick Hunt <ph...@apache.org> ha
>>>>>>>>> scritto:
>>>>>>>>>
>>>>>>>>>> On Mon, Jul 20, 2020 at 9:47 AM Enrico Olivelli <
>>>>>>>>>> eolivelli@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Il Lun 20 Lug 2020, 18:41 Patrick Hunt <ph...@apache.org>
>>>>>>>>>>> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jul 18, 2020 at 12:20 PM Andor Molnar <
>>>>>>>>>>>> andor@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> Hi Pat,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have admin rights in the new system too and
>>>>>>>>>>>>> probably can work on
>>>>>>>>>> this
>>>>>>>>>>>> on
>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>> What’s “matrix” config? Shouldn’t we just replicate
>>>>>>>>>>>>> the same jobs on
>>>>>>>>>>> the
>>>>>>>>>>>>> new instance?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Not sure on the exact name/feature but "matrix" is
>>>>>>>>>>>> basically the
>>>>>>>>>> ability
>>>>>>>>>>> to
>>>>>>>>>>>> say "run this build/test across a set of JDK versions"
>>>>>>>>>>>> rather than a
>>>>>>>>>>> single
>>>>>>>>>>>> version. As a result, instead of 3 jobs for zk3.6.0,
>>>>>>>>>>>> jdk 1/2/3 you end
>>>>>>>>>> up
>>>>>>>>>>>> with a single job which runs three times, one for each
>>>>>>>>>>>> jdk type and
>>>>>>>>>>>> summarizes the results. I've seen this before, I assume
>>>>>>>>>>>> it's a feature
>>>>>>>>>> of
>>>>>>>>>>>> jenkins itself?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We should use Jenkins files and this configuration will
>>>>>>>>>>> be easy and
>>>>>>>>>>> committed to git
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The ability to do "gitops" would be amazing!
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes.
>>>>>>>>> But I saw on Slack that Andor already started to create new
>>>>>>>>> jobs.
>>>>>>>>> We should do only one way in order not to waste time.
>>>>>>>>>
>>>>>>>>> Andor can you please share your work?
>>>>>>>>> Thanks for doing it
>>>>>>>>>
>>>>>>>>> Enrico
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Patrick
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Enrico
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Patrick
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Andor
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2020. Jul 17., at 2:51, Patrick Hunt <
>>>>>>>>>>>>>> phunt@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated the job I linked earlier based on what's
>>>>>>>>>>>>>> the latest on
>>>>>>>>>> the
>>>>>>>>>>>>> legacy
>>>>>>>>>>>>>> jenkins. It ran successfully
>>>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/215/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I didn't replicate every config setting - main gap
>>>>>>>>>>>>>> is the spotbugs
>>>>>>>>>>>>>> post-build, which seems to be missing from the new
>>>>>>>>>>>>>> jenkins plugins.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's just maven master though. Not sure about the
>>>>>>>>>>>>>> rest. Can we do
>>>>>>>>>>>> more
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> a "matrix" config in the new system vs cloning all
>>>>>>>>>>>>>> the time?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:31 PM Patrick Hunt <
>>>>>>>>>>>>>> phunt@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:22 PM Patrick Hunt <
>>>>>>>>>>>>>>> phunt@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 12:56 PM Enrico
>>>>>>>>>>>>>>>> Olivelli <
>>>>>>>>>>>> eolivelli@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> FYI
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Fun. I do notice it says "Hadoop and related
>>>>>>>>>>>>>>>> projects have their
>>>>>>>>>>> own
>>>>>>>>>>>>>>>> migration path to follow" - any insight on
>>>>>>>>>>>>>>>> that? We are or are
>>>>>>>>>> not
>>>>>>>>>>>>> lumped
>>>>>>>>>>>>>>>> in? I would assume we are?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This (eventual migration) came up a while back
>>>>>>>>>>>>>>>> on the Hadoop PMC
>>>>>>>>>>> and
>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> volunteered to try for us (ZK). I was never
>>>>>>>>>>>>>>>> able to get it to
>>>>>>>>>>> work, I
>>>>>>>>>>>>>>>> provided feedback to infra but they never got
>>>>>>>>>>>>>>>> back, as such we
>>>>>>>>>>> have
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> project here that's not working with some basic
>>>>>>>>>>>>>>>> dependencies
>>>>>>>>>>> missing:
>>>>>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That said, we can try again. Can we verify
>>>>>>>>>>>>>>>> where ZK is supposed
>>>>>>>>>> to
>>>>>>>>>>>>> land?
>>>>>>>>>>>>>>>> Perhaps we can try to delete and recreate the
>>>>>>>>>>>>>>>> POC job I created
>>>>>>>>>> at
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> link to see if we can get it working?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see another email thread on the list saying
>>>>>>>>>>>>>>> that we are part of
>>>>>>>>>>> said
>>>>>>>>>>>>>>> "related projects". We are expected to move to
>>>>>>>>>>>>>>> http://ci-hadoop.apache.org/ within 4 weeks.
>>>>>>>>>>>>>>> Seems nodes are
>>>>>>>>>>> already
>>>>>>>>>>>>>>> being removed/migrated from the "H#" pool.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are over 400 plugins on the current
>>>>>>>>>>>>>>> builds.apache.org -
>>>>>>>>>> most
>>>>>>>>>>> of
>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> we don't need any more, or are replaced with
>>>>>>>>>>>>>>> different plugins on
>>>>>>>>>>> the
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>> system. I expect there may be some plugins we
>>>>>>>>>>>>>>> still need to
>>>>>>>>>> install
>>>>>>>>>>> to
>>>>>>>>>>>>> get
>>>>>>>>>>>>>>> you going again, which is why it is vitally
>>>>>>>>>>>>>>> important that you
>>>>>>>>>> start
>>>>>>>>>>>>>>> testing and migrating your jobs over *now*. You
>>>>>>>>>>>>>>> should all have
>>>>>>>>>>> auth.
>>>>>>>>>>>>>>> Any questions, feel free to email the
>>>>>>>>>> hadoop-migrations@apache.org
>>>>>>>>>>>>> list if
>>>>>>>>>>>>>>> you are one of the projects listed below. The
>>>>>>>>>>>>>>> rest of you, not
>>>>>>>>>>>> listed, a
>>>>>>>>>>>>>>> similar email to this one will be posted for you
>>>>>>>>>>>>>>> shortly on
>>>>>>>>>>>> builds@a.o.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> full details
>>>>>>>>>>>>>>>
>>>>>>>>
>>> https://lists.apache.org/thread.html/r21c9d40cdbf5461143dd7eb4ff48a200c2fd20c50e946884f61318fd%40%3Cbuilds.apache.org%3E
>>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>>>> Da: Gavin McDonald <gm...@apache.org>
>>>>>>>>>>>>>>>>> Date: Gio 16 Lug 2020, 18:33
>>>>>>>>>>>>>>>>> Subject: [IMPORTANT] - Migration to ci-
>>>>>>>>>>>>>>>>> builds.a.o
>>>>>>>>>>>>>>>>> To: builds <bu...@apache.org>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This NOTICE is for everyone on
>>>>>>>>>>>>>>>>> builds.apache.org. We are
>>>>>>>>>>> migrating
>>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> Cloudbees based Client Master called
>>>>>>>>>> https://ci-builds.apache.org
>>>>>>>>>>> .
>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> migrations of all jobs needs to be done
>>>>>>>>>>>>>>>>> before the switch off
>>>>>>>>>> date
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 15th
>>>>>>>>>>>>>>>>> August 2020, so you have a maximum of 4
>>>>>>>>>>>>>>>>> weeks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There is no tool or automated way of
>>>>>>>>>>>>>>>>> migrating your jobs, the
>>>>>>>>>>>>>>>>> differences in the platforms, the plugins and
>>>>>>>>>>>>>>>>> the setup makes it
>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>> to do in a safe way. So, you all need to
>>>>>>>>>>>>>>>>> start creating new jobs
>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> ci-infra.a.o and then , when you are happy,
>>>>>>>>>>>>>>>>> turn off your old
>>>>>>>>>>> builds
>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> builds.a.o.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There are currently 4 agents over there ready
>>>>>>>>>>>>>>>>> to take jobs,
>>>>>>>>>> plus a
>>>>>>>>>>>>>>>>> floating
>>>>>>>>>>>>>>>>> agent which is shared amongst many masters
>>>>>>>>>>>>>>>>> (more to come). I
>>>>>>>>>> will
>>>>>>>>>>>>> migrate
>>>>>>>>>>>>>>>>> away 2 more agents from builds.a.o to ci-
>>>>>>>>>>>>>>>>> builds.a.o every few
>>>>>>>>>>> days,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> will keep an eye of load across both and
>>>>>>>>>>>>>>>>> adjust accordingly.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If needed, create a ticket on INFRA jira for
>>>>>>>>>>>>>>>>> any issues that
>>>>>>>>>> crop
>>>>>>>>>>>> up,
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> email here on builds@a.o. there may be one or
>>>>>>>>>>>>>>>>> two plugins we
>>>>>>>>>> need
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> install/tweak etc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We will be not using 'Views' at the top
>>>>>>>>>>>>>>>>> level, but rather we
>>>>>>>>>> will
>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>> advantage of 'Folders Plus' - each project
>>>>>>>>>>>>>>>>> will get its own
>>>>>>>>>> Folder
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> authorisation access to create/edit jobs etc
>>>>>>>>>>>>>>>>> within that
>>>>>>>>>> folder. I
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> create these folders as you ask for them to
>>>>>>>>>>>>>>>>> start with. This
>>>>>>>>>> setup
>>>>>>>>>>>>> allows
>>>>>>>>>>>>>>>>> for credentials isolation amongst other
>>>>>>>>>>>>>>>>> benefits, including but
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> limited
>>>>>>>>>>>>>>>>> to exclusive agents (Controlled Agents) for
>>>>>>>>>>>>>>>>> your own use ,
>>>>>>>>>> should
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> any project targeted donations of agents.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As with other aspects of the ASF, projects
>>>>>>>>>>>>>>>>> can choose to just
>>>>>>>>>>> enable
>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> committers access to their folder, just ask.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We will re-use builds.apache.org as a CNAME
>>>>>>>>>>>>>>>>> to ci-builds.a.o
>>>>>>>>>> but
>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> be setting up any forwarding rules or
>>>>>>>>>>>>>>>>> anything like that.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, please, get started *now *on this so you
>>>>>>>>>>>>>>>>> can be sure we are
>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> completed before the final cutoff date of
>>>>>>>>>>>>>>>>> 15th August 2020.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Any questions - I expect a few (dozen :) ) -
>>>>>>>>>>>>>>>>> ask away and/or
>>>>>>>>>> file
>>>>>>>>>>>>> INFRA
>>>>>>>>>>>>>>>>> tickets.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hadoop and related projects have their own
>>>>>>>>>>>>>>>>> migration path to
>>>>>>>>>>> follow,
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> cut off date, Cassandra, Beam, CouchDB have
>>>>>>>>>>>>>>>>> already migrated and
>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>> very well in their new homes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Lets get going ...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Gavin McDonald*
>>>>>>>>>>>>>>>>> Systems Administrator
>>>>>>>>>>>>>>>>> ASF Infrastructure Team
>>>>
>>>
>>>
>