You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by "张铎(Duo Zhang)" <pa...@gmail.com> on 2022/02/05 07:54:52 UTC

Re: New dedicated Jenkins Conrtoller

Hi, Gavin, any luck on the ssh transfer plugin?

Thanks.

张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月29日周六 14:53写道：

> So any updates here Gavin?
>
> Is the ssh transfer plugin available now?
>
> Thanks.
>
> Sean Busbey <bu...@apache.org> 于2022年1月23日周日 09:04写道：
>
>> It looks like I currently do have access to manage the ci-hbase
>> controller.
>>
>>
>> Could you check that the git-websites label handling where a common
>> machine gets leased to the ci-hbase controller is set up properly? or give
>> me a pointer on how I would check this myself?
>>
>> I started trying to migrate our website generation job again and it's
>> been waiting for a node with the git-websites label to show up for ~8 hours
>> at this point.
>>
>> https://ci-hbase.apache.org/job/hbase_generate_website/3/
>>
>>
>>
>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald <gm...@apache.org>
>> wrote:
>>
>>> Hi,
>>> Timestamper plugin installed.
>>>
>>> By the way I thought I had given the PMC members permissions to manage
>>> Jenkins
>>> on ci-hbase including installing plugins etc, can someone check?
>>>
>>>
>>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> wrote:
>>>
>>> > We faced a strange issue..
>>> >
>>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: startup
>>> failed:
>>> > WorkflowScript: 29: Invalid option type "timestamps". Valid option
>>> types: [buildDiscarder, catchError, checkoutToSubdirectory,
>>> disableConcurrentBuilds, disableResume, durabilityHint,
>>> githubProjectProperty, newContainerPerStage, overrideIndexTriggers,
>>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, rateLimitBuilds,
>>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout,
>>> waitUntil, warnError, withChecks, withContext, withCredentials, withEnv,
>>> wrap, ws] @ line 29, column 5.
>>> >        timestamps()
>>> >        ^
>>> >
>>> > 1 error
>>> >
>>> >       at
>>> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
>>> >       at
>>> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
>>> >       at
>>> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
>>> >       at
>>> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
>>> >       at
>>> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
>>> >       at
>>> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
>>> >       at
>>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
>>> >       at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
>>> >       at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
>>> >       at
>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
>>> >       at
>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
>>> >       at
>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571)
>>> >       at
>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523)
>>> >       at
>>> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334)
>>> >       at
>>> hudson.model.ResourceController.execute(ResourceController.java:97)
>>> >       at hudson.model.Executor.run(Executor.java:429)
>>> >
>>> >
>>> > After googling, it seems a restart of jenkins can fix it, or we do not
>>> > have timestamper plugin installed? Mind taking a look Gavin?
>>> >
>>> > Thanks.
>>> >
>>> > 张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月22日周六 11:40写道：
>>> >
>>> >> Oh, just saw that we could do this by ourselves.
>>> >>
>>> >> Let me have a try.
>>> >>
>>> >> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月22日周六 11:18写道：
>>> >>
>>> >>> OK, so while copying you could keep the old jenkins builds?
>>> >>> That's awesome.
>>> >>>
>>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and
>>> HBase-Flaky-Tests
>>> >>> to ci-hbase too? We need the history builds to calucate the flaky
>>> test list
>>> >>> so if all the old build data can be kept then it will help us a lot.
>>> >>>
>>> >>> Thanks~
>>> >>>
>>> >>> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月22日周六 11:03写道：
>>> >>>
>>> >>>> Thank you Gavin.
>>> >>>>
>>> >>>> Let me first move some light weighted job first to see if
>>> everything is
>>> >>>> OK.
>>> >>>>
>>> >>>> HBase Nightly, some flaky related jobs, and also the main pre commit
>>> >>>> job for HBase are tied together. We need to move them at once.
>>> >>>>
>>> >>>> Gavin McDonald <gm...@apache.org> 于2022年1月22日周六 02:44写道：
>>> >>>>
>>> >>>>> Thanks for your patience
>>> >>>>>
>>> >>>>> I have 'copied' over a job -
>>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/
>>> >>>>>
>>> >>>>> which of course made me aware of what plugins I needed to install.
>>> >>>>>
>>> >>>>> Feel free to delete that, or use it and delete the old, and let me
>>> >>>>> know if have any issues
>>> >>>>> with any other missing plugins.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) <
>>> palomino219@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> I’ve also replied on the jira issue.
>>> >>>>>>
>>> >>>>>> You can try to copy a job to the new Jenkins.
>>> >>>>>>
>>> >>>>>> Thanks.
>>> >>>>>>
>>> >>>>>> Gavin McDonald <gm...@apache.org>于2022年1月20日 周四18:45写道：
>>> >>>>>>
>>> >>>>>>> Im taking a look ...
>>> >>>>>>>
>>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey <bu...@apache.org>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> > Hi Gavin
>>> >>>>>>> >
>>> >>>>>>> > I went to start migrating jobs to the ci-hbase controller, but
>>> the
>>> >>>>>>> new job
>>> >>>>>>> > workflow doesn't include pipelines or multibranch pipelines. We
>>> >>>>>>> use both of
>>> >>>>>>> > these job types for nearly all of our work. Can they be
>>> enabled?
>>> >>>>>>> >
>>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald <
>>> >>>>>>> gmcdonald@apache.org>
>>> >>>>>>> > wrote:
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >>
>>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote:
>>> >>>>>>> >> > Thank you Gavin.
>>> >>>>>>> >> >
>>> >>>>>>> >> > We have some urls in our jenkins files which references the
>>> >>>>>>> artifacts of
>>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. For
>>> >>>>>>> example,
>>> >>>>>>> >> here:
>>> >>>>>>> >> >
>>> >>>>>>> >> >
>>> >>>>>>> >>
>>> >>>>>>>
>>> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47
>>> >>>>>>> >> >
>>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file a
>>> HBase
>>> >>>>>>> issue
>>> >>>>>>> >> to do
>>> >>>>>>> >> > this.
>>> >>>>>>> >> >
>>> >>>>>>> >> > And will report any problems on INFRA_22760 while migrating.
>>> >>>>>>> When the
>>> >>>>>>> >> > migration is half done, you can move for example, 5 hbase
>>> nodes
>>> >>>>>>> from
>>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally
>>> done,
>>> >>>>>>> you can
>>> >>>>>>> >> move
>>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to ci-hbase.
>>> >>>>>>> >> >
>>> >>>>>>> >> > WDYT?
>>> >>>>>>> >>
>>> >>>>>>> >> However you want to do it is fine with me :)
>>> >>>>>>> >>
>>> >>>>>>> >> You have 10 nodes on there right now to get you started.
>>> >>>>>>> >>
>>> >>>>>>> >> Let me know how you get on
>>> >>>>>>> >>
>>> >>>>>>> >> >
>>> >>>>>>> >> > Thanks.
>>> >>>>>>> >> >
>>> >>>>>>> >> > Gavin McDonald <gm...@apache.org> 于2022年1月18日周二
>>> 16:16写道：
>>> >>>>>>> >> >
>>> >>>>>>> >> > > Hi All,
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > I created an Infra ticket for tracking.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 in
>>> >>>>>>> ci-hadoop.a.o
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > I have created for you 10 new nodes so far out of our
>>> Hetzner
>>> >>>>>>> DC
>>> >>>>>>> >> > > of a similar specifications to the hbase1-10.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Those are hbase11-20 and are available right now for
>>> testing
>>> >>>>>>> on
>>> >>>>>>> >> > > ci-hbase.apache.org
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > I do not know your workflow well enough to decide the next
>>> >>>>>>> steps for
>>> >>>>>>> >> you
>>> >>>>>>> >> > > for testing.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Options include:
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto ci-hbase
>>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of ci-hadoop
>>> onto
>>> >>>>>>> ci-hbase
>>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs over
>>> for
>>> >>>>>>> testing.
>>> >>>>>>> >> > >     - I like copy, however quite a few projects jobs would
>>> >>>>>>> not be
>>> >>>>>>> >> > > appropriate
>>> >>>>>>> >> > >       to have the same job running twice, including those
>>> that
>>> >>>>>>> >> deploy, or
>>> >>>>>>> >> > > those that
>>> >>>>>>> >> > >       build from PRs and continue a workflow to report
>>> back,
>>> >>>>>>> or
>>> >>>>>>> >> approve
>>> >>>>>>> >> > > for merge.
>>> >>>>>>> >> > >       So for a copy, would need to be a job that just
>>> 'builds
>>> >>>>>>> and
>>> >>>>>>> >> tests'
>>> >>>>>>> >> > > with no outflow.
>>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to
>>> test
>>> >>>>>>> the nodes
>>> >>>>>>> >> > > setup and the
>>> >>>>>>> >> > >     main controller setup, without affecting current jobs.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Obviously, you may to tweak version controlled script
>>> files
>>> >>>>>>> such as a
>>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder unless
>>> >>>>>>> testing from a
>>> >>>>>>> >> > > branch created for the purpose.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > I may be overthinking things, and option 1 , the simplest
>>> and
>>> >>>>>>> most
>>> >>>>>>> >> > > straight forward from my end would also suit you guys and
>>> we
>>> >>>>>>> deal
>>> >>>>>>> >> with any
>>> >>>>>>> >> > > breakage as it happens (I'm thinking of integrations with
>>> >>>>>>> other
>>> >>>>>>> >> services,
>>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that are
>>> >>>>>>> needed etc.)
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so we can
>>> get
>>> >>>>>>> using
>>> >>>>>>> >> these
>>> >>>>>>> >> > > nodes, after which I can then proceed to move hbase1-10
>>> nodes
>>> >>>>>>> over
>>> >>>>>>> >> also.
>>> >>>>>>> >> > > And then start adding more into the mix.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is
>>> >>>>>>> available on
>>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared and
>>> >>>>>>> leased out to
>>> >>>>>>> >> > > controllers as needed upon request of a job.
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Thanks
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > Gav...
>>> >>>>>>> >> > >
>>> >>>>>>> >> > >
>>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote:
>>> >>>>>>> >> > > > Thanks you Gavin.
>>> >>>>>>> >> > > >
>>> >>>>>>> >> > > > You can do it in your convenience, just let me know so I
>>> >>>>>>> could
>>> >>>>>>> >> change the
>>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at least
>>> our
>>> >>>>>>> pre
>>> >>>>>>> >> commit job
>>> >>>>>>> >> > > > still uses the Hadoop label.
>>> >>>>>>> >> > > >
>>> >>>>>>> >> > > > And another thing is about the hbase_generate_website
>>> job,
>>> >>>>>>> it needs
>>> >>>>>>> >> to be
>>> >>>>>>> >> > > > executed on the nodes under the git-websites label. I'm
>>> not
>>> >>>>>>> sure if
>>> >>>>>>> >> the
>>> >>>>>>> >> > > > label is still available after the migration.
>>> >>>>>>> >> > > >
>>> >>>>>>> >> > > > Gavin McDonald <gm...@apache.org> 于2021年12月8日周三
>>> >>>>>>> 17:28写道：
>>> >>>>>>> >> > > >
>>> >>>>>>> >> > > > > Hi HBase devs.
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > This is to let you know that Infra has created a
>>> >>>>>>> dedicated Jenkins
>>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org .
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > You have new incoming donated agents which will get
>>> >>>>>>> connected to
>>> >>>>>>> >> this
>>> >>>>>>> >> > > new
>>> >>>>>>> >> > > > > controller by Infra.
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > You also have 10 or so agents connected to
>>> >>>>>>> >> > > https://ci-hadoop.apache.org -
>>> >>>>>>> >> > > > > I
>>> >>>>>>> >> > > > > would like to move these agents - and all of your
>>> jobs -
>>> >>>>>>> over to
>>> >>>>>>> >> the
>>> >>>>>>> >> > > new
>>> >>>>>>> >> > > > > controller as soon as possible.
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > You may or may not need to reconfigure your jobs and
>>> or
>>> >>>>>>> >> JenkinsFile or
>>> >>>>>>> >> > > > > other.
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > Can I get the go ahead to move these at my
>>> convenience or
>>> >>>>>>> suggest
>>> >>>>>>> >> > > please a
>>> >>>>>>> >> > > > > timeline for Infra to move these agents.
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > Thanks!
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > --
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > > > *Gavin McDonald*
>>> >>>>>>> >> > > > > Systems Administrator
>>> >>>>>>> >> > > > > ASF Infrastructure Team
>>> >>>>>>> >> > > > >
>>> >>>>>>> >> > > >
>>> >>>>>>> >> > >
>>> >>>>>>> >> >
>>> >>>>>>> >>
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>>
>>> >>>>>>> *Gavin McDonald*
>>> >>>>>>> Systems Administrator
>>> >>>>>>> ASF Infrastructure Team
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>>
>>> >>>>> *Gavin McDonald*
>>> >>>>> Systems Administrator
>>> >>>>> ASF Infrastructure Team
>>> >>>>>
>>> >>>>
>>>
>>> --
>>>
>>> *Gavin McDonald*
>>> Systems Administrator
>>> ASF Infrastructure Team
>>>
>>