You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Zoltan Haindrich <ki...@rxd.hu> on 2020/07/31 16:06:21 UTC
Re: maintainers of hadoop-multibranch please limit concurrent runs
Hey All!
For Hive we also use the multi-branch pipeline plugin [1] - and in the last couple months I've seen some issues with it.
Probably most importantly w.r.t to the current issue: the hadoop-multibranch job right now is configured "merge the PR with the master" which is a handy feature - but it
has a backstab: in case branch indexing happens - it will re-discover all PRs as well in case the master has new commits!
As a quick fix to this I would recommend to disable the periodic scanning (which is set to 30 minutes right now).
Second: it is hard to limit the number of active jobs of a multi-branch pipeline job ; for Hive we use resource locking [2] to ensure that at most 5 runs are executing at
any given time.
I've cc-ed the hadoop-common mailing list.
cheers,
Zoltan
[1] http://ci.hive.apache.org/job/hive-precommit/
[2] https://plugins.jenkins.io/lockable-resources/
On 7/31/20 5:35 PM, Sean Busbey wrote:
> would the maintainers of
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/
>
> please configure it to limit the number of concurrent executions and clean up the current backlog? y'all have saturate all workers for the last 10 hours and it is not apparent if the running jobs can actually complete.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: maintainers of hadoop-multibranch please limit concurrent runs
Posted by Akira Ajisaka <aa...@apache.org>.
All the ci-hadoop users,
I apologize for starving the resources by my misconfiguration.
Aborted all the hadoop-multibranch jobs and modified the configuration as
follows:
- Disabled the periodic scanning.
- Set "Discover pull requests from origin/forks" to "The current pull
request revision"
Thank you Sean for the report and thanks Zoltan for your advice.
Best regards,
Akira
On Sat, Aug 1, 2020 at 1:07 AM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey All!
>
> For Hive we also use the multi-branch pipeline plugin [1] - and in the
> last couple months I've seen some issues with it.
>
> Probably most importantly w.r.t to the current issue: the
> hadoop-multibranch job right now is configured "merge the PR with the
> master" which is a handy feature - but it
> has a backstab: in case branch indexing happens - it will re-discover all
> PRs as well in case the master has new commits!
> As a quick fix to this I would recommend to disable the periodic scanning
> (which is set to 30 minutes right now).
>
> Second: it is hard to limit the number of active jobs of a multi-branch
> pipeline job ; for Hive we use resource locking [2] to ensure that at most
> 5 runs are executing at
> any given time.
>
> I've cc-ed the hadoop-common mailing list.
>
> cheers,
> Zoltan
>
> [1] http://ci.hive.apache.org/job/hive-precommit/
> [2] https://plugins.jenkins.io/lockable-resources/
>
>
> On 7/31/20 5:35 PM, Sean Busbey wrote:
> > would the maintainers of
> >
> > https://ci-hadoop.apache.org/job/hadoop-multibranch/
> >
> > please configure it to limit the number of concurrent executions and
> clean up the current backlog? y'all have saturate all workers for the last
> 10 hours and it is not apparent if the running jobs can actually complete.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: maintainers of hadoop-multibranch please limit concurrent runs
Posted by Akira Ajisaka <aa...@apache.org>.
All the ci-hadoop users,
I apologize for starving the resources by my misconfiguration.
Aborted all the hadoop-multibranch jobs and modified the configuration as
follows:
- Disabled the periodic scanning.
- Set "Discover pull requests from origin/forks" to "The current pull
request revision"
Thank you Sean for the report and thanks Zoltan for your advice.
Best regards,
Akira
On Sat, Aug 1, 2020 at 1:07 AM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey All!
>
> For Hive we also use the multi-branch pipeline plugin [1] - and in the
> last couple months I've seen some issues with it.
>
> Probably most importantly w.r.t to the current issue: the
> hadoop-multibranch job right now is configured "merge the PR with the
> master" which is a handy feature - but it
> has a backstab: in case branch indexing happens - it will re-discover all
> PRs as well in case the master has new commits!
> As a quick fix to this I would recommend to disable the periodic scanning
> (which is set to 30 minutes right now).
>
> Second: it is hard to limit the number of active jobs of a multi-branch
> pipeline job ; for Hive we use resource locking [2] to ensure that at most
> 5 runs are executing at
> any given time.
>
> I've cc-ed the hadoop-common mailing list.
>
> cheers,
> Zoltan
>
> [1] http://ci.hive.apache.org/job/hive-precommit/
> [2] https://plugins.jenkins.io/lockable-resources/
>
>
> On 7/31/20 5:35 PM, Sean Busbey wrote:
> > would the maintainers of
> >
> > https://ci-hadoop.apache.org/job/hadoop-multibranch/
> >
> > please configure it to limit the number of concurrent executions and
> clean up the current backlog? y'all have saturate all workers for the last
> 10 hours and it is not apparent if the running jobs can actually complete.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>