You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Lisen Mu <im...@gmail.com> on 2013/05/29 08:21:45 UTC
random failure on tests
Hello,
I'm setting up a jenkins job for hbase, building branch 0.95 (from github)
under jdk 6.
However sometimes the build pasts, sometimes does not. Several tests are
likely to fail, such as:
org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs
org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak
yet they do not fail every time.
Any clue about what might be the problem? Thanks.
>From the last failed build:
the executed mvn command line:
Executing Maven: -B -f
/var/lib/jenkins/jobs/HBase-0.95-jdk-6/workspace/pom.xml clean package
Possibly related log:
org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
Error Message
Index: 1, Size: 1
Stacktrace
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpc(TestDelayedRpc.java:112)
at org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue(TestDelayedRpc.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Re: random failure on tests
Posted by Nicolas Liochon <nk...@gmail.com>.
Yes for the stack size. See as well the config 2.1.2.5, nofile & nproc are
important for the tests.
Check as well with the resource checker (see chapter 15.7.3.7) that the
values actually used in the tests execution are the ones that you expect.
You want MaxFileDescriptor to be above 2K to be safe. We run the tests with
a value set to 65K. Same for the nproc, in linux it actually controls the
number of java thread you can create.
On Wed, May 29, 2013 at 12:02 PM, Lisen Mu <im...@gmail.com> wrote:
> Nicolas,
>
> Thanks for the reply!
>
> I run small & medium tests only, and I have 8G ram on the build server. My
> passing rate is also around 50%, but I'm not running -PrunAllTests.
>
> I can find 2 differences between official build and mine: missing prebuild
> step 'rm -rf /tmp/hbase-jenkins/hbase' and stack size is 1024 vs 8192 in
> ulimit. Could this be a problem? I would try out anyway. Thanks!
>
>
>
>
>
> On Wed, May 29, 2013 at 3:12 PM, Nicolas Liochon <nk...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Option 1:
> > We still have some flaky tests. You can benchmark you build against
> > https://builds.apache.org/job/HBase-TRUNK/ and
> > https://builds.apache.org/job/hbase-0.95/
> > You can also use this tool: https://github.com/jeffreyz88/jenkins-toolsto
> > get a review on the last fails:
> >
> > On 0.95, we have some tests that failed ~15% of the time:
> >
> > org.apache.hadoop.hbase.client.testadmin.testforcesplitmultifamily
> > 80/6/13 1 1 1 1 1 1 1 1 1 1 1 1
> > -1 0 -1
> > org.apache.hadoop.hbase.client.testhcm.testdeleteforzkconnleak 86/6/6
> > 1 -1 0 1 1 1 1 1 1 1 1 1 1 1 1
> > org.apache.hadoop.hbase.client.testmultiparallel.testactivethreadscount
> > 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> > 1 1
> >
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
> > 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> > 1 1
> >
> >
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
> > 86/6/6 1 1 1 1 1 1 1 1 1 1 -1 0 1
> > 1 1
> > org.apache.hadoop.hbase.rest.client.testremoteadmin.testclusterstatus
> > 86/6/6 1 1 1 1 1 1 1 1 1 1 1 1 -1
> > 0 1
> >
> >
> org.apache.hadoop.hbase.security.access.testaccesscontroller.testglobalauthorizationfornewregisteredrs
> > 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> > 1 1
> > org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
> > 86/6/6
> >
> >
> > Option 2:
> > You've got some issues in you env. We run the test in parallel (5 b
> default
> > when you run all tests, 2 when you run only the small & medium ones). 5
> > requires around 10 GB or memory. If you have less or if the built is
> > shared, you may enter into strange conditions around test timing
> > requirements.
> > You also need to use oracle jdk.
> >
> > See as well http://hbase.apache.org/book.html / 15.7.3. Running tests,
> for
> > extra parameters.
> >
> >
> > These options are not exclusive. It seems that these days trunk build is
> ok
> > ~80% of the time these days, and 0.95 50%. You should expect something
> > similar.
> >
> >
> > A test becomes flaky because a patch breaks it just a little. The patch
> > passes the peer review and the precommit runs, but after a while the
> > randomness shows up, and we need to fix the code again. It's a never
> ending
> > story. Any help in fixing them is always greatly appreciated.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Wed, May 29, 2013 at 8:21 AM, Lisen Mu <im...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > I'm setting up a jenkins job for hbase, building branch 0.95 (from
> > github)
> > > under jdk 6.
> > >
> > > However sometimes the build pasts, sometimes does not. Several tests
> are
> > > likely to fail, such as:
> > >
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> > > org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs
> > > org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak
> > >
> > > yet they do not fail every time.
> > >
> > > Any clue about what might be the problem? Thanks.
> > >
> > >
> > > From the last failed build:
> > >
> > > the executed mvn command line:
> > >
> > > Executing Maven: -B -f
> > > /var/lib/jenkins/jobs/HBase-0.95-jdk-6/workspace/pom.xml clean package
> > >
> > >
> > >
> > > Possibly related log:
> > >
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> > >
> > > Error Message
> > >
> > > Index: 1, Size: 1
> > >
> > > Stacktrace
> > >
> > > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> > > at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> > > at java.util.ArrayList.get(ArrayList.java:322)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpc(TestDelayedRpc.java:112)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue(TestDelayedRpc.java:71)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> > > at
> > >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > > at
> > >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> > > at
> > >
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > > at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> > > at
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> > > at
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > > at
> > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> > > at
> > > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> > > at
> > org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> > > at
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> > > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> > > at org.junit.runners.Suite.runChild(Suite.java:127)
> > > at org.junit.runners.Suite.runChild(Suite.java:26)
> > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > > at
> > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> >
>
Re: random failure on tests
Posted by Lisen Mu <im...@gmail.com>.
Nicolas,
Thanks for the reply!
I run small & medium tests only, and I have 8G ram on the build server. My
passing rate is also around 50%, but I'm not running -PrunAllTests.
I can find 2 differences between official build and mine: missing prebuild
step 'rm -rf /tmp/hbase-jenkins/hbase' and stack size is 1024 vs 8192 in
ulimit. Could this be a problem? I would try out anyway. Thanks!
On Wed, May 29, 2013 at 3:12 PM, Nicolas Liochon <nk...@gmail.com> wrote:
> Hello,
>
> Option 1:
> We still have some flaky tests. You can benchmark you build against
> https://builds.apache.org/job/HBase-TRUNK/ and
> https://builds.apache.org/job/hbase-0.95/
> You can also use this tool: https://github.com/jeffreyz88/jenkins-tools to
> get a review on the last fails:
>
> On 0.95, we have some tests that failed ~15% of the time:
>
> org.apache.hadoop.hbase.client.testadmin.testforcesplitmultifamily
> 80/6/13 1 1 1 1 1 1 1 1 1 1 1 1
> -1 0 -1
> org.apache.hadoop.hbase.client.testhcm.testdeleteforzkconnleak 86/6/6
> 1 -1 0 1 1 1 1 1 1 1 1 1 1 1 1
> org.apache.hadoop.hbase.client.testmultiparallel.testactivethreadscount
> 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> 1 1
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
> 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> 1 1
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
> 86/6/6 1 1 1 1 1 1 1 1 1 1 -1 0 1
> 1 1
> org.apache.hadoop.hbase.rest.client.testremoteadmin.testclusterstatus
> 86/6/6 1 1 1 1 1 1 1 1 1 1 1 1 -1
> 0 1
>
> org.apache.hadoop.hbase.security.access.testaccesscontroller.testglobalauthorizationfornewregisteredrs
> 86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
> 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
> 86/6/6
>
>
> Option 2:
> You've got some issues in you env. We run the test in parallel (5 b default
> when you run all tests, 2 when you run only the small & medium ones). 5
> requires around 10 GB or memory. If you have less or if the built is
> shared, you may enter into strange conditions around test timing
> requirements.
> You also need to use oracle jdk.
>
> See as well http://hbase.apache.org/book.html / 15.7.3. Running tests, for
> extra parameters.
>
>
> These options are not exclusive. It seems that these days trunk build is ok
> ~80% of the time these days, and 0.95 50%. You should expect something
> similar.
>
>
> A test becomes flaky because a patch breaks it just a little. The patch
> passes the peer review and the precommit runs, but after a while the
> randomness shows up, and we need to fix the code again. It's a never ending
> story. Any help in fixing them is always greatly appreciated.
>
> Cheers,
>
> Nicolas
>
>
> On Wed, May 29, 2013 at 8:21 AM, Lisen Mu <im...@gmail.com> wrote:
>
> > Hello,
> >
> > I'm setting up a jenkins job for hbase, building branch 0.95 (from
> github)
> > under jdk 6.
> >
> > However sometimes the build pasts, sometimes does not. Several tests are
> > likely to fail, such as:
> >
> >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> > org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs
> > org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak
> >
> > yet they do not fail every time.
> >
> > Any clue about what might be the problem? Thanks.
> >
> >
> > From the last failed build:
> >
> > the executed mvn command line:
> >
> > Executing Maven: -B -f
> > /var/lib/jenkins/jobs/HBase-0.95-jdk-6/workspace/pom.xml clean package
> >
> >
> >
> > Possibly related log:
> >
> >
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> >
> > Error Message
> >
> > Index: 1, Size: 1
> >
> > Stacktrace
> >
> > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> > at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> > at java.util.ArrayList.get(ArrayList.java:322)
> > at
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpc(TestDelayedRpc.java:112)
> > at
> >
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue(TestDelayedRpc.java:71)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> > at
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> > at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> > at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> > at org.junit.runners.Suite.runChild(Suite.java:127)
> > at org.junit.runners.Suite.runChild(Suite.java:26)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
>
Re: random failure on tests
Posted by Nicolas Liochon <nk...@gmail.com>.
Hello,
Option 1:
We still have some flaky tests. You can benchmark you build against
https://builds.apache.org/job/HBase-TRUNK/ and
https://builds.apache.org/job/hbase-0.95/
You can also use this tool: https://github.com/jeffreyz88/jenkins-tools to
get a review on the last fails:
On 0.95, we have some tests that failed ~15% of the time:
org.apache.hadoop.hbase.client.testadmin.testforcesplitmultifamily
80/6/13 1 1 1 1 1 1 1 1 1 1 1 1
-1 0 -1
org.apache.hadoop.hbase.client.testhcm.testdeleteforzkconnleak 86/6/6
1 -1 0 1 1 1 1 1 1 1 1 1 1 1 1
org.apache.hadoop.hbase.client.testmultiparallel.testactivethreadscount
86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
1 1
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
1 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
86/6/6 1 1 1 1 1 1 1 1 1 1 -1 0 1
1 1
org.apache.hadoop.hbase.rest.client.testremoteadmin.testclusterstatus
86/6/6 1 1 1 1 1 1 1 1 1 1 1 1 -1
0 1
org.apache.hadoop.hbase.security.access.testaccesscontroller.testglobalauthorizationfornewregisteredrs
86/6/6 1 1 1 1 1 1 1 -1 0 1 1 1 1
1 1
org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
86/6/6
Option 2:
You've got some issues in you env. We run the test in parallel (5 b default
when you run all tests, 2 when you run only the small & medium ones). 5
requires around 10 GB or memory. If you have less or if the built is
shared, you may enter into strange conditions around test timing
requirements.
You also need to use oracle jdk.
See as well http://hbase.apache.org/book.html / 15.7.3. Running tests, for
extra parameters.
These options are not exclusive. It seems that these days trunk build is ok
~80% of the time these days, and 0.95 50%. You should expect something
similar.
A test becomes flaky because a patch breaks it just a little. The patch
passes the peer review and the precommit runs, but after a while the
randomness shows up, and we need to fix the code again. It's a never ending
story. Any help in fixing them is always greatly appreciated.
Cheers,
Nicolas
On Wed, May 29, 2013 at 8:21 AM, Lisen Mu <im...@gmail.com> wrote:
> Hello,
>
> I'm setting up a jenkins job for hbase, building branch 0.95 (from github)
> under jdk 6.
>
> However sometimes the build pasts, sometimes does not. Several tests are
> likely to fail, such as:
>
>
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs
> org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak
>
> yet they do not fail every time.
>
> Any clue about what might be the problem? Thanks.
>
>
> From the last failed build:
>
> the executed mvn command line:
>
> Executing Maven: -B -f
> /var/lib/jenkins/jobs/HBase-0.95-jdk-6/workspace/pom.xml clean package
>
>
>
> Possibly related log:
>
>
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
>
> Error Message
>
> Index: 1, Size: 1
>
> Stacktrace
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpc(TestDelayedRpc.java:112)
> at
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue(TestDelayedRpc.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>