You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Stamatis Zampetakis <za...@gmail.com> on 2019/06/03 09:00:01 UTC
[DISCUSS] Parallel parameterized unit tests
Hi all,
The past few days we had observed various problems on Jenkins (and
elsewhere) which made our test suite hang.
Haisheng and Ruben found out that the problem was the parallel execution of
parameterized tests [1]. I see two ways to unblock this situation and avoid
such problems in the future:
* disable parallel execution of unit tests;
* forbid the use of @RunWith(Parameterized.class) annotation;
at least till [1] is resolved.
What do you think?
[1] https://issues.apache.org/jira/browse/SUREFIRE-1430
Re: Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
Looks like we already used https://github.com/stephenc/jcip-annotations in pom.xml, I will keep pom.xml untouched.
- Haisheng Yuan------------------------------------------------------------------
发件人：Julian Hyde<jh...@apache.org>
日 期：2019年06月05日 02:39:43
收件人：dev@calcite.apache.org<de...@calcite.apache.org>
主 题：Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Be sure to use a version of jcip-annotations that has an acceptable license.
https://github.com/stephenc/jcip-annotations seems suitable; the
original, https://search.maven.org/artifact/net.jcip/jcip-annotations/1.0/jar,
does not.
On Tue, Jun 4, 2019 at 11:36 AM Haisheng Yuan <h....@alibaba-inc.com> wrote:
>
> Yes, I will do it.
>
> - Haisheng Yuan
>
> ------------------------------------------------------------------
> 发件人：Julian Hyde<jh...@apache.org>
> 日 期：2019年06月05日 02:23:24
> 收件人：dev@calcite.apache.org<de...@calcite.apache.org>
> 抄 送：Stamatis Zampetakis<za...@gmail.com>
> 主 题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> I think we have quorum. Can someone please commit this?
>
> Do we need to log a JIRA case to remind us to remove this workaround in future?
>
> On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L <ru...@gmail.com> wrote:
> >
> > I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> > agree, this seems the cleanest solution.
> >
> > Le mar. 4 juin 2019 à 19:54, Haisheng Yuan <h....@alibaba-inc.com> a
> > écrit :
> >
> > > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > > suggested, it works for me. This might be the least change for us with
> > > little impact. So what is the plan? Can we incorporate this change before
> > > 1.20 release? Currently I have to explicitly disable/annotate the
> > > parameterized test to be able to pass the whole test suite.
> > >
> > > - Haisheng
> > > Yuan------------------------------------------------------------------
> > > 发件人：Stamatis Zampetakis<za...@gmail.com>
> > > 日 期：2019年06月04日 07:13:00
> > > 收件人：<de...@calcite.apache.org>
> > > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > It seems that we can disable parallel execution selectively for certain
> > > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > > Maybe this can do the trick for parameterized tests while we wait for the
> > > permanent fix.
> > >
> > >
> > >
> > > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
> > > wrote:
> > >
> > > > I am inclining to change threadCount to 4, then see how it works in CI,
> > > at
> > > > least it solves the problem on my laptop.
> > > >
> > > > If the problem still shows up in CI, or it solves the problem but makes
> > > > the CI test much slower, I would rather tear up the parameterized test
> > > case.
> > > >
> > > > - Haisheng
> > > > Yuan------------------------------------------------------------------
> > > > 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> > > > 日 期：2019年06月04日 02:28:40
> > > > 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> > > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > No, my laptop has 2 CPUs, it still shows up.
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > ------------------------------------------------------------------
> > > > 发件人：Julian Hyde<jh...@apache.org>
> > > > 日 期：2019年06月04日 01:56:56
> > > > 收件人：dev<de...@calcite.apache.org>
> > > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > > machines we use for everyday development have multiple CPUs. So, could we
> > > > add a parameter so that we can turn off parallel tests only in CI?
> > > >
> > > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > > workaround.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > thanks for opening this discussion, Stamatis.
> > > > >
> > > > > I think there might be a third way. We could try to check if tuning
> > > > > surefire configuration solves the problem.
> > > > > Right now on calcite's pom.xml file we have:
> > > > >
> > > > > <surefire.parallel>both</surefire.parallel>
> > > > > <surefire.threadCount>1</surefire.threadCount>
> > > > > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> > > > >
> > > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > > to 4 seems to be a workaround, at least on some environments.
> > > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > > so maybe we should try some other values and see if that solves the
> > > > > problem without disabling parallelization.
> > > > >
> > > > > [1]
> > > >
> > > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > > >
> > > > >
> > > > >
> > > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > > > > écrit :
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> The past few days we had observed various problems on Jenkins (and
> > > > >> elsewhere) which made our test suite hang.
> > > > >>
> > > > >> Haisheng and Ruben found out that the problem was the parallel
> > > > execution of
> > > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > > avoid
> > > > >> such problems in the future:
> > > > >>
> > > > >> * disable parallel execution of unit tests;
> > > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > > >>
> > > > >> at least till [1] is resolved.
> > > > >>
> > > > >> What do you think?
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > > >>
> > > >
> > > >
> > >
> > >
Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Julian Hyde <jh...@apache.org>.
Be sure to use a version of jcip-annotations that has an acceptable license.
https://github.com/stephenc/jcip-annotations seems suitable; the
original, https://search.maven.org/artifact/net.jcip/jcip-annotations/1.0/jar,
does not.
On Tue, Jun 4, 2019 at 11:36 AM Haisheng Yuan <h....@alibaba-inc.com> wrote:
>
> Yes, I will do it.
>
> - Haisheng Yuan
>
> ------------------------------------------------------------------
> 发件人：Julian Hyde<jh...@apache.org>
> 日 期：2019年06月05日 02:23:24
> 收件人：dev@calcite.apache.org<de...@calcite.apache.org>
> 抄 送：Stamatis Zampetakis<za...@gmail.com>
> 主 题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> I think we have quorum. Can someone please commit this?
>
> Do we need to log a JIRA case to remind us to remove this workaround in future?
>
> On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L <ru...@gmail.com> wrote:
> >
> > I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> > agree, this seems the cleanest solution.
> >
> > Le mar. 4 juin 2019 à 19:54, Haisheng Yuan <h....@alibaba-inc.com> a
> > écrit :
> >
> > > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > > suggested, it works for me. This might be the least change for us with
> > > little impact. So what is the plan? Can we incorporate this change before
> > > 1.20 release? Currently I have to explicitly disable/annotate the
> > > parameterized test to be able to pass the whole test suite.
> > >
> > > - Haisheng
> > > Yuan------------------------------------------------------------------
> > > 发件人：Stamatis Zampetakis<za...@gmail.com>
> > > 日 期：2019年06月04日 07:13:00
> > > 收件人：<de...@calcite.apache.org>
> > > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > It seems that we can disable parallel execution selectively for certain
> > > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > > Maybe this can do the trick for parameterized tests while we wait for the
> > > permanent fix.
> > >
> > >
> > >
> > > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
> > > wrote:
> > >
> > > > I am inclining to change threadCount to 4, then see how it works in CI,
> > > at
> > > > least it solves the problem on my laptop.
> > > >
> > > > If the problem still shows up in CI, or it solves the problem but makes
> > > > the CI test much slower, I would rather tear up the parameterized test
> > > case.
> > > >
> > > > - Haisheng
> > > > Yuan------------------------------------------------------------------
> > > > 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> > > > 日 期：2019年06月04日 02:28:40
> > > > 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> > > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > No, my laptop has 2 CPUs, it still shows up.
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > ------------------------------------------------------------------
> > > > 发件人：Julian Hyde<jh...@apache.org>
> > > > 日 期：2019年06月04日 01:56:56
> > > > 收件人：dev<de...@calcite.apache.org>
> > > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > > machines we use for everyday development have multiple CPUs. So, could we
> > > > add a parameter so that we can turn off parallel tests only in CI?
> > > >
> > > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > > workaround.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > thanks for opening this discussion, Stamatis.
> > > > >
> > > > > I think there might be a third way. We could try to check if tuning
> > > > > surefire configuration solves the problem.
> > > > > Right now on calcite's pom.xml file we have:
> > > > >
> > > > > <surefire.parallel>both</surefire.parallel>
> > > > > <surefire.threadCount>1</surefire.threadCount>
> > > > > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> > > > >
> > > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > > to 4 seems to be a workaround, at least on some environments.
> > > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > > so maybe we should try some other values and see if that solves the
> > > > > problem without disabling parallelization.
> > > > >
> > > > > [1]
> > > >
> > > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > > >
> > > > >
> > > > >
> > > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > > > > écrit :
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> The past few days we had observed various problems on Jenkins (and
> > > > >> elsewhere) which made our test suite hang.
> > > > >>
> > > > >> Haisheng and Ruben found out that the problem was the parallel
> > > > execution of
> > > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > > avoid
> > > > >> such problems in the future:
> > > > >>
> > > > >> * disable parallel execution of unit tests;
> > > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > > >>
> > > > >> at least till [1] is resolved.
> > > > >>
> > > > >> What do you think?
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > > >>
> > > >
> > > >
> > >
> > >
Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
Yes, I will do it.
- Haisheng Yuan------------------------------------------------------------------
发件人：Julian Hyde<jh...@apache.org>
日 期：2019年06月05日 02:23:24
收件人：dev@calcite.apache.org<de...@calcite.apache.org>
抄 送：Stamatis Zampetakis<za...@gmail.com>
主 题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
I think we have quorum. Can someone please commit this?
Do we need to log a JIRA case to remind us to remove this workaround in future?
On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L <ru...@gmail.com> wrote:
>
> I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> agree, this seems the cleanest solution.
>
> Le mar. 4 juin 2019 à 19:54, Haisheng Yuan <h....@alibaba-inc.com> a
> écrit :
>
> > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > suggested, it works for me. This might be the least change for us with
> > little impact. So what is the plan? Can we incorporate this change before
> > 1.20 release? Currently I have to explicitly disable/annotate the
> > parameterized test to be able to pass the whole test suite.
> >
> > - Haisheng
> > Yuan------------------------------------------------------------------
> > 发件人：Stamatis Zampetakis<za...@gmail.com>
> > 日 期：2019年06月04日 07:13:00
> > 收件人：<de...@calcite.apache.org>
> > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > It seems that we can disable parallel execution selectively for certain
> > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > Maybe this can do the trick for parameterized tests while we wait for the
> > permanent fix.
> >
> >
> >
> > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
> > wrote:
> >
> > > I am inclining to change threadCount to 4, then see how it works in CI,
> > at
> > > least it solves the problem on my laptop.
> > >
> > > If the problem still shows up in CI, or it solves the problem but makes
> > > the CI test much slower, I would rather tear up the parameterized test
> > case.
> > >
> > > - Haisheng
> > > Yuan------------------------------------------------------------------
> > > 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> > > 日 期：2019年06月04日 02:28:40
> > > 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > No, my laptop has 2 CPUs, it still shows up.
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > ------------------------------------------------------------------
> > > 发件人：Julian Hyde<jh...@apache.org>
> > > 日 期：2019年06月04日 01:56:56
> > > 收件人：dev<de...@calcite.apache.org>
> > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > machines we use for everyday development have multiple CPUs. So, could we
> > > add a parameter so that we can turn off parallel tests only in CI?
> > >
> > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > workaround.
> > >
> > > Julian
> > >
> > >
> > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > thanks for opening this discussion, Stamatis.
> > > >
> > > > I think there might be a third way. We could try to check if tuning
> > > > surefire configuration solves the problem.
> > > > Right now on calcite's pom.xml file we have:
> > > >
> > > > <surefire.parallel>both</surefire.parallel>
> > > > <surefire.threadCount>1</surefire.threadCount>
> > > > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> > > >
> > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > to 4 seems to be a workaround, at least on some environments.
> > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > so maybe we should try some other values and see if that solves the
> > > > problem without disabling parallelization.
> > > >
> > > > [1]
> > >
> > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > >
> > > >
> > > >
> > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > > > écrit :
> > > >
> > > >> Hi all,
> > > >>
> > > >> The past few days we had observed various problems on Jenkins (and
> > > >> elsewhere) which made our test suite hang.
> > > >>
> > > >> Haisheng and Ruben found out that the problem was the parallel
> > > execution of
> > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > avoid
> > > >> such problems in the future:
> > > >>
> > > >> * disable parallel execution of unit tests;
> > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > >>
> > > >> at least till [1] is resolved.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > >>
> > >
> > >
> >
> >
Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Julian Hyde <jh...@apache.org>.
I think we have quorum. Can someone please commit this?
Do we need to log a JIRA case to remind us to remove this workaround in future?
On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L <ru...@gmail.com> wrote:
>
> I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> agree, this seems the cleanest solution.
>
> Le mar. 4 juin 2019 à 19:54, Haisheng Yuan <h....@alibaba-inc.com> a
> écrit :
>
> > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > suggested, it works for me. This might be the least change for us with
> > little impact. So what is the plan? Can we incorporate this change before
> > 1.20 release? Currently I have to explicitly disable/annotate the
> > parameterized test to be able to pass the whole test suite.
> >
> > - Haisheng
> > Yuan------------------------------------------------------------------
> > 发件人：Stamatis Zampetakis<za...@gmail.com>
> > 日 期：2019年06月04日 07:13:00
> > 收件人：<de...@calcite.apache.org>
> > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > It seems that we can disable parallel execution selectively for certain
> > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > Maybe this can do the trick for parameterized tests while we wait for the
> > permanent fix.
> >
> >
> >
> > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
> > wrote:
> >
> > > I am inclining to change threadCount to 4, then see how it works in CI,
> > at
> > > least it solves the problem on my laptop.
> > >
> > > If the problem still shows up in CI, or it solves the problem but makes
> > > the CI test much slower, I would rather tear up the parameterized test
> > case.
> > >
> > > - Haisheng
> > > Yuan------------------------------------------------------------------
> > > 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> > > 日 期：2019年06月04日 02:28:40
> > > 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > No, my laptop has 2 CPUs, it still shows up.
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > ------------------------------------------------------------------
> > > 发件人：Julian Hyde<jh...@apache.org>
> > > 日 期：2019年06月04日 01:56:56
> > > 收件人：dev<de...@calcite.apache.org>
> > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > machines we use for everyday development have multiple CPUs. So, could we
> > > add a parameter so that we can turn off parallel tests only in CI?
> > >
> > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > workaround.
> > >
> > > Julian
> > >
> > >
> > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > thanks for opening this discussion, Stamatis.
> > > >
> > > > I think there might be a third way. We could try to check if tuning
> > > > surefire configuration solves the problem.
> > > > Right now on calcite's pom.xml file we have:
> > > >
> > > > <surefire.parallel>both</surefire.parallel>
> > > > <surefire.threadCount>1</surefire.threadCount>
> > > > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> > > >
> > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > to 4 seems to be a workaround, at least on some environments.
> > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > so maybe we should try some other values and see if that solves the
> > > > problem without disabling parallelization.
> > > >
> > > > [1]
> > >
> > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > >
> > > >
> > > >
> > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > > > écrit :
> > > >
> > > >> Hi all,
> > > >>
> > > >> The past few days we had observed various problems on Jenkins (and
> > > >> elsewhere) which made our test suite hang.
> > > >>
> > > >> Haisheng and Ruben found out that the problem was the parallel
> > > execution of
> > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > avoid
> > > >> such problems in the future:
> > > >>
> > > >> * disable parallel execution of unit tests;
> > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > >>
> > > >> at least till [1] is resolved.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > >>
> > >
> > >
> >
> >
Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Ruben Q L <ru...@gmail.com>.
I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
agree, this seems the cleanest solution.
Le mar. 4 juin 2019 à 19:54, Haisheng Yuan <h....@alibaba-inc.com> a
écrit :
> I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> suggested, it works for me. This might be the least change for us with
> little impact. So what is the plan? Can we incorporate this change before
> 1.20 release? Currently I have to explicitly disable/annotate the
> parameterized test to be able to pass the whole test suite.
>
> - Haisheng
> Yuan------------------------------------------------------------------
> 发件人：Stamatis Zampetakis<za...@gmail.com>
> 日 期：2019年06月04日 07:13:00
> 收件人：<de...@calcite.apache.org>
> 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> It seems that we can disable parallel execution selectively for certain
> test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> Maybe this can do the trick for parameterized tests while we wait for the
> permanent fix.
>
>
>
> On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
> wrote:
>
> > I am inclining to change threadCount to 4, then see how it works in CI,
> at
> > least it solves the problem on my laptop.
> >
> > If the problem still shows up in CI, or it solves the problem but makes
> > the CI test much slower, I would rather tear up the parameterized test
> case.
> >
> > - Haisheng
> > Yuan------------------------------------------------------------------
> > 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> > 日 期：2019年06月04日 02:28:40
> > 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> >
> > No, my laptop has 2 CPUs, it still shows up.
> >
> > Thanks ~
> > Haisheng Yuan
> > ------------------------------------------------------------------
> > 发件人：Julian Hyde<jh...@apache.org>
> > 日 期：2019年06月04日 01:56:56
> > 收件人：dev<de...@calcite.apache.org>
> > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> >
> > Am I correct that this problem only shows up on machines with 1 CPU?
> >
> > Such machines exist in virtualization environments (e.g. Travis) but the
> > machines we use for everyday development have multiple CPUs. So, could we
> > add a parameter so that we can turn off parallel tests only in CI?
> >
> > My goal is to avoid a drop in speed and test coverage while we wait for a
> > workaround.
> >
> > Julian
> >
> >
> > > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > thanks for opening this discussion, Stamatis.
> > >
> > > I think there might be a third way. We could try to check if tuning
> > > surefire configuration solves the problem.
> > > Right now on calcite's pom.xml file we have:
> > >
> > > <surefire.parallel>both</surefire.parallel>
> > > <surefire.threadCount>1</surefire.threadCount>
> > > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> > >
> > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > to 4 seems to be a workaround, at least on some environments.
> > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > so maybe we should try some other values and see if that solves the
> > > problem without disabling parallelization.
> > >
> > > [1]
> >
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > >
> > >
> > >
> > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > > écrit :
> > >
> > >> Hi all,
> > >>
> > >> The past few days we had observed various problems on Jenkins (and
> > >> elsewhere) which made our test suite hang.
> > >>
> > >> Haisheng and Ruben found out that the problem was the parallel
> > execution of
> > >> parameterized tests [1]. I see two ways to unblock this situation and
> > avoid
> > >> such problems in the future:
> > >>
> > >> * disable parallel execution of unit tests;
> > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > >>
> > >> at least till [1] is resolved.
> > >>
> > >> What do you think?
> > >>
> > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > >>
> >
> >
>
>
Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis suggested, it works for me. This might be the least change for us with little impact. So what is the plan? Can we incorporate this change before 1.20 release? Currently I have to explicitly disable/annotate the parameterized test to be able to pass the whole test suite.
- Haisheng Yuan------------------------------------------------------------------
发件人：Stamatis Zampetakis<za...@gmail.com>
日 期：2019年06月04日 07:13:00
收件人：<de...@calcite.apache.org>
主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
It seems that we can disable parallel execution selectively for certain
test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
Maybe this can do the trick for parameterized tests while we wait for the
permanent fix.
On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
wrote:
> I am inclining to change threadCount to 4, then see how it works in CI, at
> least it solves the problem on my laptop.
>
> If the problem still shows up in CI, or it solves the problem but makes
> the CI test much slower, I would rather tear up the parameterized test case.
>
> - Haisheng
> Yuan------------------------------------------------------------------
> 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> 日 期：2019年06月04日 02:28:40
> 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> > Am I correct that this problem only shows up on machines with 1 CPU?
>
> No, my laptop has 2 CPUs, it still shows up.
>
> Thanks ~
> Haisheng Yuan
> ------------------------------------------------------------------
> 发件人：Julian Hyde<jh...@apache.org>
> 日 期：2019年06月04日 01:56:56
> 收件人：dev<de...@calcite.apache.org>
> 主 题：Re: [DISCUSS] Parallel parameterized unit tests
>
> Am I correct that this problem only shows up on machines with 1 CPU?
>
> Such machines exist in virtualization environments (e.g. Travis) but the
> machines we use for everyday development have multiple CPUs. So, could we
> add a parameter so that we can turn off parallel tests only in CI?
>
> My goal is to avoid a drop in speed and test coverage while we wait for a
> workaround.
>
> Julian
>
>
> > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> >
> > Hi all,
> >
> > thanks for opening this discussion, Stamatis.
> >
> > I think there might be a third way. We could try to check if tuning
> > surefire configuration solves the problem.
> > Right now on calcite's pom.xml file we have:
> >
> > <surefire.parallel>both</surefire.parallel>
> > <surefire.threadCount>1</surefire.threadCount>
> > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> >
> > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > to 4 seems to be a workaround, at least on some environments.
> > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > so maybe we should try some other values and see if that solves the
> > problem without disabling parallelization.
> >
> > [1]
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> >
> >
> >
> > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > écrit :
> >
> >> Hi all,
> >>
> >> The past few days we had observed various problems on Jenkins (and
> >> elsewhere) which made our test suite hang.
> >>
> >> Haisheng and Ruben found out that the problem was the parallel
> execution of
> >> parameterized tests [1]. I see two ways to unblock this situation and
> avoid
> >> such problems in the future:
> >>
> >> * disable parallel execution of unit tests;
> >> * forbid the use of @RunWith(Parameterized.class) annotation;
> >>
> >> at least till [1] is resolved.
> >>
> >> What do you think?
> >>
> >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> >>
>
>
Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Stamatis Zampetakis <za...@gmail.com>.
It seems that we can disable parallel execution selectively for certain
test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
Maybe this can do the trick for parameterized tests while we wait for the
permanent fix.
On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan <h....@alibaba-inc.com>
wrote:
> I am inclining to change threadCount to 4, then see how it works in CI, at
> least it solves the problem on my laptop.
>
> If the problem still shows up in CI, or it solves the problem but makes
> the CI test much slower, I would rather tear up the parameterized test case.
>
> - Haisheng
> Yuan------------------------------------------------------------------
> 发件人：Haisheng Yuan<h....@alibaba-inc.com>
> 日 期：2019年06月04日 02:28:40
> 收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
> 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> > Am I correct that this problem only shows up on machines with 1 CPU?
>
> No, my laptop has 2 CPUs, it still shows up.
>
> Thanks ~
> Haisheng Yuan
> ------------------------------------------------------------------
> 发件人：Julian Hyde<jh...@apache.org>
> 日 期：2019年06月04日 01:56:56
> 收件人：dev<de...@calcite.apache.org>
> 主 题：Re: [DISCUSS] Parallel parameterized unit tests
>
> Am I correct that this problem only shows up on machines with 1 CPU?
>
> Such machines exist in virtualization environments (e.g. Travis) but the
> machines we use for everyday development have multiple CPUs. So, could we
> add a parameter so that we can turn off parallel tests only in CI?
>
> My goal is to avoid a drop in speed and test coverage while we wait for a
> workaround.
>
> Julian
>
>
> > On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
> >
> > Hi all,
> >
> > thanks for opening this discussion, Stamatis.
> >
> > I think there might be a third way. We could try to check if tuning
> > surefire configuration solves the problem.
> > Right now on calcite's pom.xml file we have:
> >
> > <surefire.parallel>both</surefire.parallel>
> > <surefire.threadCount>1</surefire.threadCount>
> > <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
> >
> > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > to 4 seems to be a workaround, at least on some environments.
> > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > so maybe we should try some other values and see if that solves the
> > problem without disabling parallelization.
> >
> > [1]
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> >
> >
> >
> > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> > écrit :
> >
> >> Hi all,
> >>
> >> The past few days we had observed various problems on Jenkins (and
> >> elsewhere) which made our test suite hang.
> >>
> >> Haisheng and Ruben found out that the problem was the parallel
> execution of
> >> parameterized tests [1]. I see two ways to unblock this situation and
> avoid
> >> such problems in the future:
> >>
> >> * disable parallel execution of unit tests;
> >> * forbid the use of @RunWith(Parameterized.class) annotation;
> >>
> >> at least till [1] is resolved.
> >>
> >> What do you think?
> >>
> >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> >>
>
>
Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
I am inclining to change threadCount to 4, then see how it works in CI, at least it solves the problem on my laptop.
If the problem still shows up in CI, or it solves the problem but makes the CI test much slower, I would rather tear up the parameterized test case.
- Haisheng Yuan------------------------------------------------------------------
发件人：Haisheng Yuan<h....@alibaba-inc.com>
日 期：2019年06月04日 02:28:40
收件人：Julian Hyde<jh...@apache.org>; dev<de...@calcite.apache.org>
主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> Am I correct that this problem only shows up on machines with 1 CPU?
No, my laptop has 2 CPUs, it still shows up.
Thanks ~
Haisheng Yuan
------------------------------------------------------------------
发件人：Julian Hyde<jh...@apache.org>
日 期：2019年06月04日 01:56:56
收件人：dev<de...@calcite.apache.org>
主 题：Re: [DISCUSS] Parallel parameterized unit tests
Am I correct that this problem only shows up on machines with 1 CPU?
Such machines exist in virtualization environments (e.g. Travis) but the machines we use for everyday development have multiple CPUs. So, could we add a parameter so that we can turn off parallel tests only in CI?
My goal is to avoid a drop in speed and test coverage while we wait for a workaround.
Julian
> On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
>
> Hi all,
>
> thanks for opening this discussion, Stamatis.
>
> I think there might be a third way. We could try to check if tuning
> surefire configuration solves the problem.
> Right now on calcite's pom.xml file we have:
>
> <surefire.parallel>both</surefire.parallel>
> <surefire.threadCount>1</surefire.threadCount>
> <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
>
> Thanks to Haisheng's tests, we know that changing threadCount from 1
> to 4 seems to be a workaround, at least on some environments.
> Moreover, I have noticed that parallel "both" value is deprecated [1],
> so maybe we should try some other values and see if that solves the
> problem without disabling parallelization.
>
> [1] http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
>
>
>
> Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> écrit :
>
>> Hi all,
>>
>> The past few days we had observed various problems on Jenkins (and
>> elsewhere) which made our test suite hang.
>>
>> Haisheng and Ruben found out that the problem was the parallel execution of
>> parameterized tests [1]. I see two ways to unblock this situation and avoid
>> such problems in the future:
>>
>> * disable parallel execution of unit tests;
>> * forbid the use of @RunWith(Parameterized.class) annotation;
>>
>> at least till [1] is resolved.
>>
>> What do you think?
>>
>> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
>>
Re: Re: [DISCUSS] Parallel parameterized unit tests
Posted by Haisheng Yuan <h....@alibaba-inc.com>.
> Am I correct that this problem only shows up on machines with 1 CPU?
No, my laptop has 2 CPUs, it still shows up.
Thanks ~
Haisheng Yuan
------------------------------------------------------------------
发件人：Julian Hyde<jh...@apache.org>
日 期：2019年06月04日 01:56:56
收件人：dev<de...@calcite.apache.org>
主 题：Re: [DISCUSS] Parallel parameterized unit tests
Am I correct that this problem only shows up on machines with 1 CPU?
Such machines exist in virtualization environments (e.g. Travis) but the machines we use for everyday development have multiple CPUs. So, could we add a parameter so that we can turn off parallel tests only in CI?
My goal is to avoid a drop in speed and test coverage while we wait for a workaround.
Julian
> On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
>
> Hi all,
>
> thanks for opening this discussion, Stamatis.
>
> I think there might be a third way. We could try to check if tuning
> surefire configuration solves the problem.
> Right now on calcite's pom.xml file we have:
>
> <surefire.parallel>both</surefire.parallel>
> <surefire.threadCount>1</surefire.threadCount>
> <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
>
> Thanks to Haisheng's tests, we know that changing threadCount from 1
> to 4 seems to be a workaround, at least on some environments.
> Moreover, I have noticed that parallel "both" value is deprecated [1],
> so maybe we should try some other values and see if that solves the
> problem without disabling parallelization.
>
> [1] http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
>
>
>
> Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> écrit :
>
>> Hi all,
>>
>> The past few days we had observed various problems on Jenkins (and
>> elsewhere) which made our test suite hang.
>>
>> Haisheng and Ruben found out that the problem was the parallel execution of
>> parameterized tests [1]. I see two ways to unblock this situation and avoid
>> such problems in the future:
>>
>> * disable parallel execution of unit tests;
>> * forbid the use of @RunWith(Parameterized.class) annotation;
>>
>> at least till [1] is resolved.
>>
>> What do you think?
>>
>> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
>>
Re: [DISCUSS] Parallel parameterized unit tests
Posted by Julian Hyde <jh...@apache.org>.
Am I correct that this problem only shows up on machines with 1 CPU?
Such machines exist in virtualization environments (e.g. Travis) but the machines we use for everyday development have multiple CPUs. So, could we add a parameter so that we can turn off parallel tests only in CI?
My goal is to avoid a drop in speed and test coverage while we wait for a workaround.
Julian
> On Jun 3, 2019, at 2:29 AM, Ruben Q L <ru...@gmail.com> wrote:
>
> Hi all,
>
> thanks for opening this discussion, Stamatis.
>
> I think there might be a third way. We could try to check if tuning
> surefire configuration solves the problem.
> Right now on calcite's pom.xml file we have:
>
> <surefire.parallel>both</surefire.parallel>
> <surefire.threadCount>1</surefire.threadCount>
> <surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
>
> Thanks to Haisheng's tests, we know that changing threadCount from 1
> to 4 seems to be a workaround, at least on some environments.
> Moreover, I have noticed that parallel "both" value is deprecated [1],
> so maybe we should try some other values and see if that solves the
> problem without disabling parallelization.
>
> [1] http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
>
>
>
> Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
> écrit :
>
>> Hi all,
>>
>> The past few days we had observed various problems on Jenkins (and
>> elsewhere) which made our test suite hang.
>>
>> Haisheng and Ruben found out that the problem was the parallel execution of
>> parameterized tests [1]. I see two ways to unblock this situation and avoid
>> such problems in the future:
>>
>> * disable parallel execution of unit tests;
>> * forbid the use of @RunWith(Parameterized.class) annotation;
>>
>> at least till [1] is resolved.
>>
>> What do you think?
>>
>> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
>>
Re: [DISCUSS] Parallel parameterized unit tests
Posted by Ruben Q L <ru...@gmail.com>.
Hi all,
thanks for opening this discussion, Stamatis.
I think there might be a third way. We could try to check if tuning
surefire configuration solves the problem.
Right now on calcite's pom.xml file we have:
<surefire.parallel>both</surefire.parallel>
<surefire.threadCount>1</surefire.threadCount>
<surefire.perCoreThreadCount>true</surefire.perCoreThreadCount>
Thanks to Haisheng's tests, we know that changing threadCount from 1
to 4 seems to be a workaround, at least on some environments.
Moreover, I have noticed that parallel "both" value is deprecated [1],
so maybe we should try some other values and see if that solves the
problem without disabling parallelization.
[1] http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis <za...@gmail.com> a
écrit :
> Hi all,
>
> The past few days we had observed various problems on Jenkins (and
> elsewhere) which made our test suite hang.
>
> Haisheng and Ruben found out that the problem was the parallel execution of
> parameterized tests [1]. I see two ways to unblock this situation and avoid
> such problems in the future:
>
> * disable parallel execution of unit tests;
> * forbid the use of @RunWith(Parameterized.class) annotation;
>
> at least till [1] is resolved.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
>