You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/06/10 14:50:21 UTC
[GitHub] [hadoop] iwasakims opened a new pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
iwasakims opened a new pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065
Adding MiniMRYarnCluster configuration to
* make JHS optional
* set timeout duration to launch JHS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438320637
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
##########
@@ -259,7 +267,10 @@ public void run() {
};
}.start();
Review comment:
Should this `historyServer` and thread only be initialized when the new configuration is enabled? Would be nice to avoid allocating the resources if we don't need them.
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
##########
@@ -259,7 +267,10 @@ public void run() {
};
}.start();
- GenericTestUtils.waitFor(() -> jhsStarted, 1500, 60_000);
+ final int launchTimeout = getConfig().getInt(
+ JHAdminConfig.MR_HISTORY_MINICLUSTER_LAUNCH_TIMEOUT_MS,
+ JHAdminConfig.DEFAULT_MR_HISTORY_MINICLUSTER_LAUNCH_TIMEOUT_MS);
+ GenericTestUtils.waitFor(() -> jhsStarted, 1500, launchTimeout);
Review comment:
nit: this code would be simpler using a [`CountDownLatch`](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r442550539
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRBringup.java
##########
@@ -36,5 +40,21 @@ public void testBringUp() throws IOException {
if (mr != null) { mr.shutdown(); }
}
}
-
+
+ @Test
+ public void testMiniMRYarnClusterWithoutJHS() throws IOException {
+ MiniMRYarnCluster mr = null;
+ try {
+ final Configuration conf = new Configuration();
+ conf.setBoolean(MiniMRYarnCluster.MR_HISTORY_MINICLUSTER_ENABLED, false);
+ mr = new MiniMRYarnCluster("testMiniMRYarnClusterWithoutJHS");
+ mr.init(conf);
+ Assert.assertEquals(null, mr.getHistoryServer());
Review comment:
Sure. It makes more stable for future change. I updated the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#issuecomment-646403787
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 53s | trunk passed |
| +1 :green_heart: | compile | 1m 55s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 16s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 0s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 51s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 46s | the patch passed |
| +1 :green_heart: | javac | 1m 46s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 34s | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44) |
| +1 :green_heart: | mvnsite | 1m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 43s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
| +1 :green_heart: | findbugs | 2m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 6m 47s | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 0m 59s | hadoop-mapreduce-client-common in the patch passed. |
| +1 :green_heart: | unit | 127m 23s | hadoop-mapreduce-client-jobclient in the patch passed. |
| -1 :x: | asflicense | 0m 46s | The patch generated 1 ASF License warnings. |
| | | 204m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2065 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 91851636a1ed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d50e93ce7b6 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/3/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/3/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1579 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438521302
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1945,6 +1945,18 @@
<description>The HistoryStorage class to use to cache history data.</description>
</property>
+<property>
+ <name>mapreduce.jobhistory.minicluster.enabled</name>
+ <value>true</value>
+ <description>Whether to launch jobhistory server inminicluster.</description>
+</property>
+
+<property>
+ <name>mapreduce.jobhistory.minicluster.launch.timeout.ms</name>
+ <value>60000</value>
+ <description>Timeout to launch jobhistory server inminicluster.</description>
+</property>
+
Review comment:
We do not publish test-only configs usually. While some mini cluster configs are exposed for use cases outside unit tests (I guess), I prefer test class specific configs here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r442401950
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRBringup.java
##########
@@ -36,5 +40,21 @@ public void testBringUp() throws IOException {
if (mr != null) { mr.shutdown(); }
}
}
-
+
+ @Test
+ public void testMiniMRYarnClusterWithoutJHS() throws IOException {
+ MiniMRYarnCluster mr = null;
+ try {
+ final Configuration conf = new Configuration();
+ conf.setBoolean(MiniMRYarnCluster.MR_HISTORY_MINICLUSTER_ENABLED, false);
+ mr = new MiniMRYarnCluster("testMiniMRYarnClusterWithoutJHS");
+ mr.init(conf);
+ Assert.assertEquals(null, mr.getHistoryServer());
Review comment:
should this assert happen after the call to `start()`?
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
##########
@@ -259,7 +267,10 @@ public void run() {
};
}.start();
Review comment:
Yes, preserving the default value makes sense. I'm just wondering if the resource allocation can be skipped entirely when the `jhs` is disabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438523858
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
##########
@@ -259,7 +267,10 @@ public void run() {
};
}.start();
- GenericTestUtils.waitFor(() -> jhsStarted, 1500, 60_000);
+ final int launchTimeout = getConfig().getInt(
+ JHAdminConfig.MR_HISTORY_MINICLUSTER_LAUNCH_TIMEOUT_MS,
+ JHAdminConfig.DEFAULT_MR_HISTORY_MINICLUSTER_LAUNCH_TIMEOUT_MS);
+ GenericTestUtils.waitFor(() -> jhsStarted, 1500, launchTimeout);
Review comment:
I removed the jhsStarted var in a bit different way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] saintstack commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438319331
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1945,6 +1945,18 @@
<description>The HistoryStorage class to use to cache history data.</description>
</property>
+<property>
+ <name>mapreduce.jobhistory.minicluster.enabled</name>
+ <value>true</value>
+ <description>Whether to launch jobhistory server inminicluster.</description>
+</property>
+
+<property>
+ <name>mapreduce.jobhistory.minicluster.launch.timeout.ms</name>
+ <value>60000</value>
+ <description>Timeout to launch jobhistory server inminicluster.</description>
+</property>
+
Review comment:
These are test-time configs. Is it common practice adding them to the default xml? Just wondering.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#issuecomment-651470472
Thanks. I merged this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#issuecomment-642449988
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 3s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 59s | trunk passed |
| +1 :green_heart: | compile | 1m 56s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 43s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | trunk passed |
| +0 :ok: | spotbugs | 0m 45s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 44s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 47s | the patch passed |
| +1 :green_heart: | javac | 1m 47s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 35s | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44) |
| +1 :green_heart: | mvnsite | 1m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | the patch passed |
| +1 :green_heart: | findbugs | 2m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 6m 47s | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 1m 1s | hadoop-mapreduce-client-common in the patch passed. |
| +1 :green_heart: | unit | 126m 41s | hadoop-mapreduce-client-jobclient in the patch passed. |
| -1 :x: | asflicense | 0m 55s | The patch generated 1 ASF License warnings. |
| | | 204m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2065 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 37cd583374f6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 93b121a9717 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/2/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/2/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1617 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#issuecomment-645126492
@saintstack @ndimiduk Are you ok with the fix based on comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#issuecomment-642180042
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 8s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 10s | trunk passed |
| +1 :green_heart: | compile | 1m 55s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 47s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 16s | trunk passed |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 44s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | the patch passed |
| +1 :green_heart: | compile | 1m 44s | the patch passed |
| +1 :green_heart: | javac | 1m 44s | the patch passed |
| -0 :warning: | checkstyle | 0m 34s | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new + 43 unchanged - 1 fixed = 44 total (was 44) |
| +1 :green_heart: | mvnsite | 1m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 30s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
| +1 :green_heart: | findbugs | 3m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 31s | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 1m 0s | hadoop-mapreduce-client-common in the patch passed. |
| +1 :green_heart: | unit | 131m 14s | hadoop-mapreduce-client-jobclient in the patch passed. |
| -1 :x: | asflicense | 0m 45s | The patch generated 1 ASF License warnings. |
| | | 210m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2065 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1344c11935dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b735a777178 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/1/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/1/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/1/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1576 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2065/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438322822
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1945,6 +1945,18 @@
<description>The HistoryStorage class to use to cache history data.</description>
</property>
+<property>
+ <name>mapreduce.jobhistory.minicluster.enabled</name>
+ <value>true</value>
+ <description>Whether to launch jobhistory server inminicluster.</description>
+</property>
+
+<property>
+ <name>mapreduce.jobhistory.minicluster.launch.timeout.ms</name>
+ <value>60000</value>
+ <description>Timeout to launch jobhistory server inminicluster.</description>
+</property>
+
Review comment:
These look like production-facing configs to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims merged pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #2065: MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #2065:
URL: https://github.com/apache/hadoop/pull/2065#discussion_r438522222
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
##########
@@ -259,7 +267,10 @@ public void run() {
};
}.start();
Review comment:
historyServer is not initialized if `mapreduce.jobhistory.minicluster.enabled` is set to `false`. I would like to keep the default value to be `true` to avoid breaking existing tests of hadoop and downstreams.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org