You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/02/27 20:10:35 UTC
[GitHub] [hadoop] touchida opened a new pull request #2726: HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically
touchida opened a new pull request #2726:
URL: https://github.com/apache/hadoop/pull/2726
The PR solves the flakiness of the unit test as follows.
1. Set dfs.datanode.balance.bandwidthPerSec to 4MB so that an iteration could not move any block.
2. Set maxIdleIteratons to 1 for the iteration to report NO_MOVE_PROGERSS when there is not block move.
3. Assert that the reported status is NO_MOVE_PROGERSS, since a 10MB block cannot be moved with 4MB/s bandwidth in 2 seconds (i.e, 4MB/s * 2s = 8MB < 10MB).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2726: HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #2726:
URL: https://github.com/apache/hadoop/pull/2726
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] touchida commented on pull request #2726: HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically
Posted by GitBox <gi...@apache.org>.
touchida commented on pull request #2726:
URL: https://github.com/apache/hadoop/pull/2726#issuecomment-787475633
Both failed and crashed tests are unrelated to this PR. I created [HDFS-15864](https://issues.apache.org/jira/browse/HDFS-15864) for the former, but didn't for the latter, since I'm not sure which caused the crash, the build environment or the test itself.
Ready for review.
- Failed test
```
[ERROR] Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 47.231 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
[ERROR] testDnRestartWithHardLink(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl) Time elapsed: 1.452 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testDnRestartWithHardLink(TestFsDatasetImpl.java:1377)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
```
- Crashed test
```
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Command was /bin/sh -c cd /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -DminiClusterDedicatedDirs=true -jar /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefirebooter6106358806783121604.jar /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs/target/surefire 2021-02-27T21-36-41_009-jvmRun2 surefire4678365300915584400tmp surefire_5963761349348601015446tmp
Error occurred in starting fork, check output in log
Process Exit Code: 1
Crashed tests:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:511)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:458)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:299)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:247)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1149)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:991)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:837)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Command was /bin/sh -c cd /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -DminiClusterDedicatedDirs=true -jar /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefirebooter6106358806783121604.jar /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2726/src/hadoop-hdfs-project/hadoop-hdfs/target/surefire 2021-02-27T21-36-41_009-jvmRun2 surefire4678365300915584400tmp surefire_5963761349348601015446tmp
Error occurred in starting fork, check output in log
Process Exit Code: 1
Crashed tests:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:670)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:116)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:445)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:421)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2726: HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2726:
URL: https://github.com/apache/hadoop/pull/2726#issuecomment-789442307
@jojochuang Would you review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org