You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Flavio Junqueira <fp...@apache.org> on 2021/01/25 12:51:00 UTC
Test failures on 3.7.0 RC1 (was: Re: [VOTE] Apache ZooKeeper release
3.7.0 candidate 1)
I don't want to mess up with the vote thread, so I'm responding to this separately. I have been trying to build locally too unsuccessfully. I've been trying on an Ubuntu VM, Java 8 (build 1.8.0_181-b13), Maven 3.6.0. The set of tests failing varies from build to build, if it makes sense, I can try to collect all test failures I have seen and post.
-Flavio
> On 25 Jan 2021, at 13:38, Szalay-Bekő Máté <sz...@gmail.com> wrote:
>
> +0 (and not even binding :) )
>
> - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using OpenJDK
> 8u265 and maven 3.6.3.
> - I also built and executed unit tests for zkpython
> - the unit tests passed for the C-client and for python client
> - checkstyle and spotbugs passed
> - apache-rat passed
> - owasp (CVE check) passed
> - fatjar built (-Pfatjar)
> - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2. (using
> https://github.com/symat/zk-rolling-upgrade-test)
>
> for some reason the java unit tests failed for me.
>
> On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests executed
> successfully, but then the maven job still failed for hbase-server test
> with error message (with -DforkCount=4 and even with -DforkCount=1) like:
> -------------
> [ERROR] ExecutionException There was an error in the forked process
> [ERROR] unable to create new native thread
> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException There was an error in the forked process
> [ERROR] unable to create new native thread
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> -------------
>
>
> Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265 and
> maven 3.6.3) and I got other kinds of strange maven errors:
> -------------------
> [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd
> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server &&
> /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m
> -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3
> -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar
> -jar
> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar
> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire
> 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp
> surefire_37800399112966511000tmp
> [ERROR] Process Exit Code: 0
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
> [ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [ERROR] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [ERROR] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [ERROR] at java.lang.Thread.run(Thread.java:748)
> [ERROR]
> --------------------------------------
>
>
> These issues might be very well specific to my local (mac or docker on mac)
> environments. This is why I didn't vote with -1
> Can someone else run the java unit tests successfully locally?
>
> I also tried to check if the CI was green for the last PR on 3.7.0 (
> https://github.com/apache/zookeeper/pull/1586/checks), but it looks the CI
> haven't even started to execute the tests, due to errors in the "install C
> dependencies" step.
>
> Regards,
> Mate
>
> On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org> wrote:
>
>> +1. xsum/sig verified. rat ran clean. built and dependency checks are fine.
>> Tried running some manual clusters and it was successful.
>>
>> Regards,
>>
>> Patrick
>>
>>
>> On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen <dd...@apache.org>
>> wrote:
>>
>>>
>>> Dear all,
>>>
>>> This is a second release candidate for ZooKeeper 3.7.0. Compared to
>>> RC0, it fixes a tarball generation issue, includes a description of the
>>> 'whoami' CLI command, and incorporates a contribution to ZooInspector.
>>>
>>> ZooKeeper 3.7.0 introduces a number of new features, notably:
>>>
>>> * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874);
>>>
>>> * Quota enforcement (ZOOKEEPER-3301);
>>>
>>> * Host name canonicalization in quorum SASL authentication
>>> (ZOOKEEPER-4030);
>>>
>>> * Support for BCFKS key/trust store format (ZOOKEEPER-3950);
>>>
>>> * A choice of mandatory authentication scheme(s) (ZOOKEEPER-3561);
>>>
>>> * A "whoami" API and CLI command (ZOOKEEPER-3969);
>>>
>>> * The possibility of disabling digest authentication (ZOOKEEPER-3979);
>>>
>>> * Multiple SASL "superUsers" (ZOOKEEPER-3959);
>>>
>>> * Fast-tracking of throttled requests (ZOOKEEPER-3683);
>>>
>>> * Additional security metrics (ZOOKEEPER-3978);
>>>
>>> * SASL support in the C and Perl clients (ZOOKEEPER-1112,
>>> ZOOKEEPER-3714);
>>>
>>> * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427);
>>>
>>> * Notes on how to benchmark ZooKeeper with the YCSB tool
>>> (ZOOKEEPER-3264).
>>>
>>> The release notes are available here:
>>>
>>>
>>>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html
>>>
>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617
>>>
>>> *** Please download, test and vote by January 31st 2020, 23:59 UTC+0. ***
>>>
>>> Source files:
>>>
>>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/
>>>
>>> Maven staging repo:
>>>
>>>
>>>
>> https://repository.apache.org/content/repositories/orgapachezookeeper-1066/
>>>
>>> The release candidate tag in git to be voted upon: release-3.7.0-1
>>>
>>> https://github.com/apache/zookeeper/tree/release-3.7.0-1
>>>
>>> ZooKeeper's KEYS file containing PGP keys we use to sign the release:
>>>
>>> https://www.apache.org/dist/zookeeper/KEYS
>>>
>>> The staging version of the website is:
>>>
>>>
>>>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/
>>>
>>> Should we release this candidate?
>>>
>>> Damien Diederen
>>>
>>
Re: Test failures on 3.7.0 RC1
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
> It seems forkCount is overriden by the surefire-forkcount property.
> This seems to do what you expect:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build
-Dsurefire-forkcount=1
indeed... this seems to be the case on all active branches. my mistake :)
Thanks!
Mate
On Thu, Jan 28, 2021 at 9:38 PM Damien Diederen <dd...@apache.org>
wrote:
>
> Hi Máté, all,
>
> I managed to fight my way though some Apple ID insanity to get a Mac
> running with Catalina, Java, Maven and a C compiler. This part of the
> puzzle was particularly fun:
>
> https://dd.crosstwine.com/tmp/apple-id.png
>
> Anyway.
>
> > So I prefer to execute the java unit tests using the following command
> when
> > checking RCs:
> >
> > mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build
> -DforkCount=1
>
> This command does not work, at least on Catalina—where it forks eight
> runners:
>
> % pstree 7846
> -+= 07846 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09342 administrator /bin/sh -c cd /Users/...
> | \--- 09343 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09640 administrator /bin/sh -c cd /Users/...
> | \--- 09641 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09797 administrator /bin/sh -c cd /Users/...
> | \--- 09798 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09893 administrator /bin/sh -c cd /Users/...
> | \--- 09898 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09912 administrator /bin/sh -c cd /Users/...
> | \--- 09913 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09942 administrator /bin/sh -c cd /Users/...
> | \--- 09944 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> |-+- 09955 administrator /bin/sh -c cd /Users/...
> | \--- 09956 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
> \-+- 09960 administrator /bin/sh -c cd /Users/...
> \--- 09961 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
>
> It seems forkCount is overriden by the surefire-forkcount property.
> This seems to do what you expect:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build
> -Dsurefire-forkcount=1
>
> But independently of that, I can reproduce the issue—even with a "real"
> sequential run. It does not always result in the stack trace you saw,
> but often hangs with "funny" errors such as:
>
> [WARNING] Corrupted STDOUT by directly writing to native stream in
> forked JVM 1. See FAQ web page and the dump file
> /Users/administrator/zookeeper/zookeeper-server/target/surefire-reports/2021-01-28T17-56-26_094-jvmRun1.dumpstream
>
> # Created at 2021-01-28T18:37:51.381
> Corrupted STDOUT by directly writing to native stream in forked JVM 1.
> Stream '[59.370s][warning][os,thread] Failed to start thread -
> pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize:
> 4k, detached.'.
>
> I'm planning to have a closer look soon.
>
> Cu, -D
>
> P.-S. — Enrico: I am also able to reproduce the C compilation issues you
> reported.
>
>
> --8<---------------original message------------->8---
>
> Szalay-Bekő Máté <sz...@gmail.com> writes:
> > this is not specific to 3.7, but in general:
> > from time to time I see different tests failing intermittently. I assume
> > this can be caused by other tests running in parallel. (because
> re-running
> > those tests alone usually makes them succeed)
> > So I prefer to execute the java unit tests using the following command
> when
> > checking RCs:
> >
> > mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build
> -DforkCount=1
> >
> > As far as I remember, by default the fork count is set to 8 in maven, and
> > the CI is using (or was using before the migration) -DforkCount=4.
> >
> >
> > Regards,
> > Mate
> >
> > On Mon, Jan 25, 2021 at 1:51 PM Flavio Junqueira <fp...@apache.org> wrote:
> >
> >> I don't want to mess up with the vote thread, so I'm responding to this
> >> separately. I have been trying to build locally too unsuccessfully. I've
> >> been trying on an Ubuntu VM, Java 8 (build 1.8.0_181-b13), Maven 3.6.0.
> The
> >> set of tests failing varies from build to build, if it makes sense, I
> can
> >> try to collect all test failures I have seen and post.
> >>
> >> -Flavio
> >>
> >> > On 25 Jan 2021, at 13:38, Szalay-Bekő Máté <
> szalay.beko.mate@gmail.com>
> >> wrote:
> >> >
> >> > +0 (and not even binding :) )
> >> >
> >> > - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using
> OpenJDK
> >> > 8u265 and maven 3.6.3.
> >> > - I also built and executed unit tests for zkpython
> >> > - the unit tests passed for the C-client and for python client
> >> > - checkstyle and spotbugs passed
> >> > - apache-rat passed
> >> > - owasp (CVE check) passed
> >> > - fatjar built (-Pfatjar)
> >> > - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2.
> >> (using
> >> > https://github.com/symat/zk-rolling-upgrade-test)
> >> >
> >> > for some reason the java unit tests failed for me.
> >> >
> >> > On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests
> executed
> >> > successfully, but then the maven job still failed for hbase-server
> test
> >> > with error message (with -DforkCount=4 and even with -DforkCount=1)
> like:
> >> > -------------
> >> > [ERROR] ExecutionException There was an error in the forked process
> >> > [ERROR] unable to create new native thread
> >> > [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> >> > ExecutionException There was an error in the forked process
> >> > [ERROR] unable to create new native thread
> >> > [ERROR] at
> >> >
> >>
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> >> > -------------
> >> >
> >> >
> >> > Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265
> and
> >> > maven 3.6.3) and I got other kinds of strange maven errors:
> >> > -------------------
> >> > [ERROR] Caused by:
> >> > org.apache.maven.surefire.booter.SurefireBooterForkException: The
> forked
> >> VM
> >> > terminated without properly saying goodbye. VM crash or System.exit
> >> called?
> >> > [ERROR] Command was /bin/sh -c cd
> >> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server &&
> >> > /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m
> >> > -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3
> >> >
> >>
> -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar
> >> > -jar
> >> >
> >>
> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar
> >> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire
> >> > 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp
> >> > surefire_37800399112966511000tmp
> >> > [ERROR] Process Exit Code: 0
> >> > [ERROR] at
> >> >
> >>
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
> >> > [ERROR] at
> >> >
> >>
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
> >> > [ERROR] at
> >> >
> >>
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
> >> > [ERROR] at
> >> >
> >>
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
> >> > [ERROR] at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > [ERROR] at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> > [ERROR] at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> >> > [ERROR]
> >> > --------------------------------------
> >> >
> >> >
> >> > These issues might be very well specific to my local (mac or docker on
> >> mac)
> >> > environments. This is why I didn't vote with -1
> >> > Can someone else run the java unit tests successfully locally?
> >> >
> >> > I also tried to check if the CI was green for the last PR on 3.7.0 (
> >> > https://github.com/apache/zookeeper/pull/1586/checks), but it looks
> the
> >> CI
> >> > haven't even started to execute the tests, due to errors in the
> "install
> >> C
> >> > dependencies" step.
> >> >
> >> > Regards,
> >> > Mate
> >> >
> >> > On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org>
> wrote:
> >> >
> >> >> +1. xsum/sig verified. rat ran clean. built and dependency checks are
> >> fine.
> >> >> Tried running some manual clusters and it was successful.
> >> >>
> >> >> Regards,
> >> >>
> >> >> Patrick
> >> >>
> >> >>
> >> >> On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen <
> ddiederen@apache.org>
> >> >> wrote:
> >> >>
> >> >>>
> >> >>> Dear all,
> >> >>>
> >> >>> This is a second release candidate for ZooKeeper 3.7.0. Compared to
> >> >>> RC0, it fixes a tarball generation issue, includes a description of
> the
> >> >>> 'whoami' CLI command, and incorporates a contribution to
> ZooInspector.
> >> >>>
> >> >>> ZooKeeper 3.7.0 introduces a number of new features, notably:
> >> >>>
> >> >>> * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874);
> >> >>>
> >> >>> * Quota enforcement (ZOOKEEPER-3301);
> >> >>>
> >> >>> * Host name canonicalization in quorum SASL authentication
> >> >>> (ZOOKEEPER-4030);
> >> >>>
> >> >>> * Support for BCFKS key/trust store format (ZOOKEEPER-3950);
> >> >>>
> >> >>> * A choice of mandatory authentication scheme(s) (ZOOKEEPER-3561);
> >> >>>
> >> >>> * A "whoami" API and CLI command (ZOOKEEPER-3969);
> >> >>>
> >> >>> * The possibility of disabling digest authentication
> (ZOOKEEPER-3979);
> >> >>>
> >> >>> * Multiple SASL "superUsers" (ZOOKEEPER-3959);
> >> >>>
> >> >>> * Fast-tracking of throttled requests (ZOOKEEPER-3683);
> >> >>>
> >> >>> * Additional security metrics (ZOOKEEPER-3978);
> >> >>>
> >> >>> * SASL support in the C and Perl clients (ZOOKEEPER-1112,
> >> >>> ZOOKEEPER-3714);
> >> >>>
> >> >>> * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427);
> >> >>>
> >> >>> * Notes on how to benchmark ZooKeeper with the YCSB tool
> >> >>> (ZOOKEEPER-3264).
> >> >>>
> >> >>> The release notes are available here:
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >>
> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html
> >> >>>
> >> >>>
> >> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617
> >> >>>
> >> >>> *** Please download, test and vote by January 31st 2020, 23:59
> UTC+0.
> >> ***
> >> >>>
> >> >>> Source files:
> >> >>>
> >> >>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/
> >> >>>
> >> >>> Maven staging repo:
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachezookeeper-1066/
> >> >>>
> >> >>> The release candidate tag in git to be voted upon: release-3.7.0-1
> >> >>>
> >> >>> https://github.com/apache/zookeeper/tree/release-3.7.0-1
> >> >>>
> >> >>> ZooKeeper's KEYS file containing PGP keys we use to sign the
> release:
> >> >>>
> >> >>> https://www.apache.org/dist/zookeeper/KEYS
> >> >>>
> >> >>> The staging version of the website is:
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >>
> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/
> >> >>>
> >> >>> Should we release this candidate?
> >> >>>
> >> >>> Damien Diederen
> >> >>>
> >> >>
> >>
> >>
>
Re: Test failures on 3.7.0 RC1
Posted by Damien Diederen <dd...@apache.org>.
Hi Máté, all,
I managed to fight my way though some Apple ID insanity to get a Mac
running with Catalina, Java, Maven and a C compiler. This part of the
puzzle was particularly fun:
https://dd.crosstwine.com/tmp/apple-id.png
Anyway.
> So I prefer to execute the java unit tests using the following command when
> checking RCs:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -DforkCount=1
This command does not work, at least on Catalina—where it forks eight
runners:
% pstree 7846
-+= 07846 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09342 administrator /bin/sh -c cd /Users/...
| \--- 09343 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09640 administrator /bin/sh -c cd /Users/...
| \--- 09641 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09797 administrator /bin/sh -c cd /Users/...
| \--- 09798 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09893 administrator /bin/sh -c cd /Users/...
| \--- 09898 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09912 administrator /bin/sh -c cd /Users/...
| \--- 09913 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09942 administrator /bin/sh -c cd /Users/...
| \--- 09944 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
|-+- 09955 administrator /bin/sh -c cd /Users/...
| \--- 09956 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
\-+- 09960 administrator /bin/sh -c cd /Users/...
\--- 09961 administrator /Library/Java/JavaVir.../adoptopenjdk-8...
It seems forkCount is overriden by the surefire-forkcount property.
This seems to do what you expect:
mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -Dsurefire-forkcount=1
But independently of that, I can reproduce the issue—even with a "real"
sequential run. It does not always result in the stack trace you saw,
but often hangs with "funny" errors such as:
[WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /Users/administrator/zookeeper/zookeeper-server/target/surefire-reports/2021-01-28T17-56-26_094-jvmRun1.dumpstream
# Created at 2021-01-28T18:37:51.381
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream '[59.370s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.'.
I'm planning to have a closer look soon.
Cu, -D
P.-S. — Enrico: I am also able to reproduce the C compilation issues you
reported.
--8<---------------original message------------->8---
Szalay-Bekő Máté <sz...@gmail.com> writes:
> this is not specific to 3.7, but in general:
> from time to time I see different tests failing intermittently. I assume
> this can be caused by other tests running in parallel. (because re-running
> those tests alone usually makes them succeed)
> So I prefer to execute the java unit tests using the following command when
> checking RCs:
>
> mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -DforkCount=1
>
> As far as I remember, by default the fork count is set to 8 in maven, and
> the CI is using (or was using before the migration) -DforkCount=4.
>
>
> Regards,
> Mate
>
> On Mon, Jan 25, 2021 at 1:51 PM Flavio Junqueira <fp...@apache.org> wrote:
>
>> I don't want to mess up with the vote thread, so I'm responding to this
>> separately. I have been trying to build locally too unsuccessfully. I've
>> been trying on an Ubuntu VM, Java 8 (build 1.8.0_181-b13), Maven 3.6.0. The
>> set of tests failing varies from build to build, if it makes sense, I can
>> try to collect all test failures I have seen and post.
>>
>> -Flavio
>>
>> > On 25 Jan 2021, at 13:38, Szalay-Bekő Máté <sz...@gmail.com>
>> wrote:
>> >
>> > +0 (and not even binding :) )
>> >
>> > - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using OpenJDK
>> > 8u265 and maven 3.6.3.
>> > - I also built and executed unit tests for zkpython
>> > - the unit tests passed for the C-client and for python client
>> > - checkstyle and spotbugs passed
>> > - apache-rat passed
>> > - owasp (CVE check) passed
>> > - fatjar built (-Pfatjar)
>> > - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2.
>> (using
>> > https://github.com/symat/zk-rolling-upgrade-test)
>> >
>> > for some reason the java unit tests failed for me.
>> >
>> > On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests executed
>> > successfully, but then the maven job still failed for hbase-server test
>> > with error message (with -DforkCount=4 and even with -DforkCount=1) like:
>> > -------------
>> > [ERROR] ExecutionException There was an error in the forked process
>> > [ERROR] unable to create new native thread
>> > [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
>> > ExecutionException There was an error in the forked process
>> > [ERROR] unable to create new native thread
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
>> > -------------
>> >
>> >
>> > Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265 and
>> > maven 3.6.3) and I got other kinds of strange maven errors:
>> > -------------------
>> > [ERROR] Caused by:
>> > org.apache.maven.surefire.booter.SurefireBooterForkException: The forked
>> VM
>> > terminated without properly saying goodbye. VM crash or System.exit
>> called?
>> > [ERROR] Command was /bin/sh -c cd
>> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server &&
>> > /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m
>> > -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3
>> >
>> -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar
>> > -jar
>> >
>> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar
>> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire
>> > 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp
>> > surefire_37800399112966511000tmp
>> > [ERROR] Process Exit Code: 0
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
>> > [ERROR] at
>> >
>> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
>> > [ERROR] at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > [ERROR] at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > [ERROR] at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > [ERROR] at java.lang.Thread.run(Thread.java:748)
>> > [ERROR]
>> > --------------------------------------
>> >
>> >
>> > These issues might be very well specific to my local (mac or docker on
>> mac)
>> > environments. This is why I didn't vote with -1
>> > Can someone else run the java unit tests successfully locally?
>> >
>> > I also tried to check if the CI was green for the last PR on 3.7.0 (
>> > https://github.com/apache/zookeeper/pull/1586/checks), but it looks the
>> CI
>> > haven't even started to execute the tests, due to errors in the "install
>> C
>> > dependencies" step.
>> >
>> > Regards,
>> > Mate
>> >
>> > On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org> wrote:
>> >
>> >> +1. xsum/sig verified. rat ran clean. built and dependency checks are
>> fine.
>> >> Tried running some manual clusters and it was successful.
>> >>
>> >> Regards,
>> >>
>> >> Patrick
>> >>
>> >>
>> >> On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen <dd...@apache.org>
>> >> wrote:
>> >>
>> >>>
>> >>> Dear all,
>> >>>
>> >>> This is a second release candidate for ZooKeeper 3.7.0. Compared to
>> >>> RC0, it fixes a tarball generation issue, includes a description of the
>> >>> 'whoami' CLI command, and incorporates a contribution to ZooInspector.
>> >>>
>> >>> ZooKeeper 3.7.0 introduces a number of new features, notably:
>> >>>
>> >>> * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874);
>> >>>
>> >>> * Quota enforcement (ZOOKEEPER-3301);
>> >>>
>> >>> * Host name canonicalization in quorum SASL authentication
>> >>> (ZOOKEEPER-4030);
>> >>>
>> >>> * Support for BCFKS key/trust store format (ZOOKEEPER-3950);
>> >>>
>> >>> * A choice of mandatory authentication scheme(s) (ZOOKEEPER-3561);
>> >>>
>> >>> * A "whoami" API and CLI command (ZOOKEEPER-3969);
>> >>>
>> >>> * The possibility of disabling digest authentication (ZOOKEEPER-3979);
>> >>>
>> >>> * Multiple SASL "superUsers" (ZOOKEEPER-3959);
>> >>>
>> >>> * Fast-tracking of throttled requests (ZOOKEEPER-3683);
>> >>>
>> >>> * Additional security metrics (ZOOKEEPER-3978);
>> >>>
>> >>> * SASL support in the C and Perl clients (ZOOKEEPER-1112,
>> >>> ZOOKEEPER-3714);
>> >>>
>> >>> * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427);
>> >>>
>> >>> * Notes on how to benchmark ZooKeeper with the YCSB tool
>> >>> (ZOOKEEPER-3264).
>> >>>
>> >>> The release notes are available here:
>> >>>
>> >>>
>> >>>
>> >>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html
>> >>>
>> >>>
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617
>> >>>
>> >>> *** Please download, test and vote by January 31st 2020, 23:59 UTC+0.
>> ***
>> >>>
>> >>> Source files:
>> >>>
>> >>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/
>> >>>
>> >>> Maven staging repo:
>> >>>
>> >>>
>> >>>
>> >>
>> https://repository.apache.org/content/repositories/orgapachezookeeper-1066/
>> >>>
>> >>> The release candidate tag in git to be voted upon: release-3.7.0-1
>> >>>
>> >>> https://github.com/apache/zookeeper/tree/release-3.7.0-1
>> >>>
>> >>> ZooKeeper's KEYS file containing PGP keys we use to sign the release:
>> >>>
>> >>> https://www.apache.org/dist/zookeeper/KEYS
>> >>>
>> >>> The staging version of the website is:
>> >>>
>> >>>
>> >>>
>> >>
>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/
>> >>>
>> >>> Should we release this candidate?
>> >>>
>> >>> Damien Diederen
>> >>>
>> >>
>>
>>
Re: Test failures on 3.7.0 RC1 (was: Re: [VOTE] Apache ZooKeeper
release 3.7.0 candidate 1)
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
this is not specific to 3.7, but in general:
from time to time I see different tests failing intermittently. I assume
this can be caused by other tests running in parallel. (because re-running
those tests alone usually makes them succeed)
So I prefer to execute the java unit tests using the following command when
checking RCs:
mvn test -Dsurefire.rerunFailingTestsCount=3 -fae -Pfull-build -DforkCount=1
As far as I remember, by default the fork count is set to 8 in maven, and
the CI is using (or was using before the migration) -DforkCount=4.
Regards,
Mate
On Mon, Jan 25, 2021 at 1:51 PM Flavio Junqueira <fp...@apache.org> wrote:
> I don't want to mess up with the vote thread, so I'm responding to this
> separately. I have been trying to build locally too unsuccessfully. I've
> been trying on an Ubuntu VM, Java 8 (build 1.8.0_181-b13), Maven 3.6.0. The
> set of tests failing varies from build to build, if it makes sense, I can
> try to collect all test failures I have seen and post.
>
> -Flavio
>
> > On 25 Jan 2021, at 13:38, Szalay-Bekő Máté <sz...@gmail.com>
> wrote:
> >
> > +0 (and not even binding :) )
> >
> > - I built the source code (-Pfull-build) on Ubuntu 18.04.3 using OpenJDK
> > 8u265 and maven 3.6.3.
> > - I also built and executed unit tests for zkpython
> > - the unit tests passed for the C-client and for python client
> > - checkstyle and spotbugs passed
> > - apache-rat passed
> > - owasp (CVE check) passed
> > - fatjar built (-Pfatjar)
> > - I executed a quick rolling-upgrade test from 3.5.9 and from 3.6.2.
> (using
> > https://github.com/symat/zk-rolling-upgrade-test)
> >
> > for some reason the java unit tests failed for me.
> >
> > On mac (jdk 1.8.212 and maven 3.6.3), I got all the unit tests executed
> > successfully, but then the maven job still failed for hbase-server test
> > with error message (with -DforkCount=4 and even with -DforkCount=1) like:
> > -------------
> > [ERROR] ExecutionException There was an error in the forked process
> > [ERROR] unable to create new native thread
> > [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> > ExecutionException There was an error in the forked process
> > [ERROR] unable to create new native thread
> > [ERROR] at
> >
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> > -------------
> >
> >
> > Then I tried on a dockerized environment (ubuntu 18.4, OpenJDK 8u265 and
> > maven 3.6.3) and I got other kinds of strange maven errors:
> > -------------------
> > [ERROR] Caused by:
> > org.apache.maven.surefire.booter.SurefireBooterForkException: The forked
> VM
> > terminated without properly saying goodbye. VM crash or System.exit
> called?
> > [ERROR] Command was /bin/sh -c cd
> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server &&
> > /home/symat/.sdkman/candidates/java/8.0.265-open/jre/bin/java -Xmx512m
> > -Dtest.junit.threads=8 -Dzookeeper.junit.threadid=3
> >
> -javaagent:/home/symat/.m2/repository/org/jmockit/jmockit/1.48/jmockit-1.48.jar
> > -jar
> >
> /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire/surefirebooter8828313385463488429.jar
> > /tmp/zk/apache-zookeeper-3.7.0/zookeeper-server/target/surefire
> > 2021-01-25T11-54-03_621-jvmRun3 surefire4024538135165099286tmp
> > surefire_37800399112966511000tmp
> > [ERROR] Process Exit Code: 0
> > [ERROR] at
> >
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
> > [ERROR] at
> >
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
> > [ERROR] at
> >
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
> > [ERROR] at
> >
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
> > [ERROR] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [ERROR] at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > [ERROR] at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > [ERROR] at java.lang.Thread.run(Thread.java:748)
> > [ERROR]
> > --------------------------------------
> >
> >
> > These issues might be very well specific to my local (mac or docker on
> mac)
> > environments. This is why I didn't vote with -1
> > Can someone else run the java unit tests successfully locally?
> >
> > I also tried to check if the CI was green for the last PR on 3.7.0 (
> > https://github.com/apache/zookeeper/pull/1586/checks), but it looks the
> CI
> > haven't even started to execute the tests, due to errors in the "install
> C
> > dependencies" step.
> >
> > Regards,
> > Mate
> >
> > On Sun, Jan 24, 2021 at 11:39 PM Patrick Hunt <ph...@apache.org> wrote:
> >
> >> +1. xsum/sig verified. rat ran clean. built and dependency checks are
> fine.
> >> Tried running some manual clusters and it was successful.
> >>
> >> Regards,
> >>
> >> Patrick
> >>
> >>
> >> On Sun, Jan 24, 2021 at 12:11 PM Damien Diederen <dd...@apache.org>
> >> wrote:
> >>
> >>>
> >>> Dear all,
> >>>
> >>> This is a second release candidate for ZooKeeper 3.7.0. Compared to
> >>> RC0, it fixes a tarball generation issue, includes a description of the
> >>> 'whoami' CLI command, and incorporates a contribution to ZooInspector.
> >>>
> >>> ZooKeeper 3.7.0 introduces a number of new features, notably:
> >>>
> >>> * An API to start a ZooKeeper server from Java (ZOOKEEPER-3874);
> >>>
> >>> * Quota enforcement (ZOOKEEPER-3301);
> >>>
> >>> * Host name canonicalization in quorum SASL authentication
> >>> (ZOOKEEPER-4030);
> >>>
> >>> * Support for BCFKS key/trust store format (ZOOKEEPER-3950);
> >>>
> >>> * A choice of mandatory authentication scheme(s) (ZOOKEEPER-3561);
> >>>
> >>> * A "whoami" API and CLI command (ZOOKEEPER-3969);
> >>>
> >>> * The possibility of disabling digest authentication (ZOOKEEPER-3979);
> >>>
> >>> * Multiple SASL "superUsers" (ZOOKEEPER-3959);
> >>>
> >>> * Fast-tracking of throttled requests (ZOOKEEPER-3683);
> >>>
> >>> * Additional security metrics (ZOOKEEPER-3978);
> >>>
> >>> * SASL support in the C and Perl clients (ZOOKEEPER-1112,
> >>> ZOOKEEPER-3714);
> >>>
> >>> * A new zkSnapshotComparer.sh tool (ZOOKEEPER-3427);
> >>>
> >>> * Notes on how to benchmark ZooKeeper with the YCSB tool
> >>> (ZOOKEEPER-3264).
> >>>
> >>> The release notes are available here:
> >>>
> >>>
> >>>
> >>
> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/releasenotes.html
> >>>
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12346617
> >>>
> >>> *** Please download, test and vote by January 31st 2020, 23:59 UTC+0.
> ***
> >>>
> >>> Source files:
> >>>
> >>> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/
> >>>
> >>> Maven staging repo:
> >>>
> >>>
> >>>
> >>
> https://repository.apache.org/content/repositories/orgapachezookeeper-1066/
> >>>
> >>> The release candidate tag in git to be voted upon: release-3.7.0-1
> >>>
> >>> https://github.com/apache/zookeeper/tree/release-3.7.0-1
> >>>
> >>> ZooKeeper's KEYS file containing PGP keys we use to sign the release:
> >>>
> >>> https://www.apache.org/dist/zookeeper/KEYS
> >>>
> >>> The staging version of the website is:
> >>>
> >>>
> >>>
> >>
> https://people.apache.org/~ddiederen/zookeeper-3.7.0-candidate-1/website/
> >>>
> >>> Should we release this candidate?
> >>>
> >>> Damien Diederen
> >>>
> >>
>
>