[NOTICE] The pre commit job is a bit unstable than before

["张铎(Duo Zhang)" <pa...@gmail.com>]

After switching to ci-hbase, our pre commit job runs much slower than
before.

This is because the new hbase jenkins node has less cpu cores than the
hadoop jenkins node, so we have less concurrency when executing UTs as the
default forkCount is 0.25C, which is related to the number of cores of the
build node.

In HBASE-26828, I've increased the concurrency to 0.5C, so we can almost
get the speed back, but it is also easier to hit failing flaky tests. I've
already opened several issues to address some of them, while other
committers are also doing the same thing.

Please stay tuned.

Thanks.

Re: [NOTICE] The pre commit job is a bit unstable than before

["Tak Lon (Stephen) Wu" <ta...@apache.org>]

+1 and thanks for fixing it.

-Stephen

On Wed, Mar 16, 2022 at 4:17 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> The nightly jobs are executed on the same machine with pre commits, so they
> will be effected too :)
>
> Nick Dimiduk <nd...@apache.org> 于2022年3月16日周三 18:42写道：
>
> > I think it's not just, nightlies are also more flakey than usual.
> >
> > Thank you for being proactive on this, Duo (and others)!
> >
> > On Tue, Mar 15, 2022 at 2:29 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> > > After switching to ci-hbase, our pre commit job runs much slower than
> > > before.
> > >
> > > This is because the new hbase jenkins node has less cpu cores than the
> > > hadoop jenkins node, so we have less concurrency when executing UTs as
> > the
> > > default forkCount is 0.25C, which is related to the number of cores of
> > the
> > > build node.
> > >
> > > In HBASE-26828, I've increased the concurrency to 0.5C, so we can almost
> > > get the speed back, but it is also easier to hit failing flaky tests.
> > I've
> > > already opened several issues to address some of them, while other
> > > committers are also doing the same thing.
> > >
> > > Please stay tuned.
> > >
> > > Thanks.
> > >
> >

Re: [NOTICE] The pre commit job is a bit unstable than before

["张铎(Duo Zhang)" <pa...@gmail.com>]

The nightly jobs are executed on the same machine with pre commits, so they
will be effected too :)

Nick Dimiduk <nd...@apache.org> 于2022年3月16日周三 18:42写道：

> I think it's not just, nightlies are also more flakey than usual.
>
> Thank you for being proactive on this, Duo (and others)!
>
> On Tue, Mar 15, 2022 at 2:29 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > After switching to ci-hbase, our pre commit job runs much slower than
> > before.
> >
> > This is because the new hbase jenkins node has less cpu cores than the
> > hadoop jenkins node, so we have less concurrency when executing UTs as
> the
> > default forkCount is 0.25C, which is related to the number of cores of
> the
> > build node.
> >
> > In HBASE-26828, I've increased the concurrency to 0.5C, so we can almost
> > get the speed back, but it is also easier to hit failing flaky tests.
> I've
> > already opened several issues to address some of them, while other
> > committers are also doing the same thing.
> >
> > Please stay tuned.
> >
> > Thanks.
> >
>

Re: [NOTICE] The pre commit job is a bit unstable than before

[Nick Dimiduk <nd...@apache.org>]

I think it's not just, nightlies are also more flakey than usual.

Thank you for being proactive on this, Duo (and others)!

On Tue, Mar 15, 2022 at 2:29 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> After switching to ci-hbase, our pre commit job runs much slower than
> before.
>
> This is because the new hbase jenkins node has less cpu cores than the
> hadoop jenkins node, so we have less concurrency when executing UTs as the
> default forkCount is 0.25C, which is related to the number of cores of the
> build node.
>
> In HBASE-26828, I've increased the concurrency to 0.5C, so we can almost
> get the speed back, but it is also easier to hit failing flaky tests. I've
> already opened several issues to address some of them, while other
> committers are also doing the same thing.
>
> Please stay tuned.
>
> Thanks.
>