You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/10/23 22:23:30 UTC
[GitHub] [hadoop] sunchao opened a new pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
sunchao opened a new pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411
In [HADOOP-15832](https://issues.apache.org/jira/browse/HADOOP-15832) we excluded org.bouncycastle from the shaded client jars but still relocate them to `org.apache.hadoop.shaded`. This caused downstream applications no longer able to use `MiniYARNCluster` from `hadoop-client-minicluster` since there is no way to bring their own org.bouncycastle test dependency.
```
Error: Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException
Error: at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
Error: at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
Error: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Error: at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
Error: at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:862)
Error: at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error: at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
Error: at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
Error: at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error: at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:353)
Error: at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:127)
Error: at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:488)
Error: at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error: at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
Error: at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:321)
Error: at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error: at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:94)
Error: at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
Error: at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
Error: at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
Error: at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:61)
Error: at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
Error: at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
Error: at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
Error: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Error: at java.lang.Thread.run(Thread.java:748)
```
This fixes the issue by excluding org.bouncycastle from relocation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r520073675
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
Review comment:
maybe we should include them for test-only purpose? I think hadoop-client-minicluster should always be test scope right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-716788701
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 43s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 23s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 0m 26s | | branch/hadoop-client-modules/hadoop-client-api no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-runtime no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 22s | | branch/hadoop-client-modules/hadoop-client-minicluster no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (findbugsXml.xml) |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| -0 :warning: | checkstyle | 0m 14s | [/diff-checkstyle-hadoop-client-modules.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/artifact/out/diff-checkstyle-hadoop-client-modules.txt) | hadoop-client-modules: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| -1 :x: | shadedclient | 14m 42s | | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-api has no data from findbugs |
| +0 :ok: | findbugs | 0m 20s | | hadoop-client-modules/hadoop-client-runtime has no data from findbugs |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-minicluster has no data from findbugs |
| +0 :ok: | findbugs | 0m 18s | | hadoop-client-modules/hadoop-client-integration-tests has no data from findbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-integration-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 97m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux bfc9aec29534 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 872440610f0 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/testReport/ |
| Max. process+thread count | 402 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-integration-tests U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-721941949
@jojochuang @steveloughran added a unit test for this - can you take another look? thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r520074799
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -44,6 +44,10 @@
import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
Review comment:
will fix. I was trying to follow the existing style though which separates imports based on package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-725985434
My two aliases for showing commit info; `lg` is the one I normally log with
```
# log with indicator of trust
alias.lg=log --date=relative --color --graph --pretty=format:'%Cred%h%Creset %C(blue)%G?%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold yellow)<%an>%Creset %C(cyan)<%cn>%Creset' --abbrev-commit
# log with full GPG signature infor
alias.log1s=log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative --show-signature
```
Now, if you go through history (e.g. branch-2.8) you'll come across a GPG key I had to revoke; the yubikey keygen turned out to be broken. Git doesn't really handle that well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-726242645
Thanks @steveloughran , the commands look better than mine so I'll take them. I can sign my future cherry-picks. Question though: where should I add my public key? should it added into https://dist.apache.org/repos/dist/release/hadoop/common/KEYS? sorry not very familiar with this process.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-725660820
seen this is merged in. @sunchao -you got a GPG key in KEYS yet? Be good to GPG sign your patches. Github does it for the web ui ones, but we have to do it ourselves on the cherrypicks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r520671724
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
Review comment:
it may be test only, but the other test suite may have its own configs that it wants. If we put them in the test JAR, it becomes near impossible to change. People will hate us (more)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r520092586
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
+ conf.setInt("yarn.scheduler.capacity.root.default.capacity", 100);
+ yarnCluster = new MiniYARNCluster(getClass().getName(), 1, 1, 1, 1);
+ yarnCluster.init(conf);
+ yarnCluster.start();
}
@After
public void clusterDown() {
if (cluster != null) {
cluster.close();
}
+ if (yarnCluster != null) {
Review comment:
Nice. Didn't know this util method, but seems we can only use it with `yarnCluster`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-725549736
Sure @dongjoon-hyun - updated. I'll backport this to branch-3.3 also later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-721939853
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 29m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 57s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 0m 27s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 0m 31s | | branch/hadoop-client-modules/hadoop-client-api no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 27s | | branch/hadoop-client-modules/hadoop-client-runtime no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 28s | | branch/hadoop-client-modules/hadoop-client-minicluster no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 28s | | branch/hadoop-client-modules/hadoop-client-check-test-invariants no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 27s | | branch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (findbugsXml.xml) |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | [/diff-checkstyle-hadoop-client-modules.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/3/artifact/out/diff-checkstyle-hadoop-client-modules.txt) | hadoop-client-modules: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 18s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 3s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | findbugs | 0m 21s | | hadoop-client-modules/hadoop-client-api has no data from findbugs |
| +0 :ok: | findbugs | 0m 22s | | hadoop-client-modules/hadoop-client-runtime has no data from findbugs |
| +0 :ok: | findbugs | 0m 23s | | hadoop-client-modules/hadoop-client-minicluster has no data from findbugs |
| +0 :ok: | findbugs | 0m 21s | | hadoop-client-modules/hadoop-client-check-test-invariants has no data from findbugs |
| +0 :ok: | findbugs | 0m 21s | | hadoop-client-modules/hadoop-client-integration-tests has no data from findbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 21s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | unit | 0m 21s | | hadoop-client-check-test-invariants in the patch passed. |
| +1 :green_heart: | unit | 0m 21s | | hadoop-client-integration-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 130m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle |
| uname | Linux ccf860621740 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2cb9e8f66a3 |
| Default Java | Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/3/testReport/ |
| Max. process+thread count | 721 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/3/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-725474389
Thanks! Could you update the JIRA `Fixed Version` accordingly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-715655226
> I'm not sure I understand the use case here. Is the MiniYARNCluster simply broken out of box? Or that Spark wants to test using MiniYARNCluster but substitue a custom version of BouncyCastle?
Yes I'd say it's broken out of box since 3.3.0: there are two issues, this one and [HADOOP-17327](https://issues.apache.org/jira/browse/HADOOP-17327). Spark doesn't use a custom version but since bouncycastle is relocated _and_ not shaded, there is no way for Spark to introduce the dependency, so this never works ...
> Our client jar contains only one IT ITUseMiniCluster and it checks only hdfs and webhdfs clients. How about adding an IT for MiniYARNCluster?
This is a good suggestion, thanks. I'll perhaps add it in HADOOP-17327 since without MiniYARNCluster won't work without both being resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-724306768
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 54s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 37s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 0m 23s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 0m 25s | | branch/hadoop-client-modules/hadoop-client-api no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 21s | | branch/hadoop-client-modules/hadoop-client-runtime no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-minicluster no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 21s | | branch/hadoop-client-modules/hadoop-client-check-test-invariants no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (findbugsXml.xml) |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 12m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 17m 11s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | findbugs | 0m 20s | | hadoop-client-modules/hadoop-client-api has no data from findbugs |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-runtime has no data from findbugs |
| +0 :ok: | findbugs | 0m 18s | | hadoop-client-modules/hadoop-client-minicluster has no data from findbugs |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-check-test-invariants has no data from findbugs |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-integration-tests has no data from findbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-check-test-invariants in the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-integration-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 110m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle |
| uname | Linux 6db97700c318 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1ea3f742462 |
| Default Java | Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/4/testReport/ |
| Max. process+thread count | 511 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/4/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-724796787
committed to trunk. Is this needed on branch-3.3 too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r519821934
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -44,6 +44,10 @@
import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+
+import static org.junit.Assert.assertTrue;
Review comment:
unused, according to checkstyle
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -44,6 +44,10 @@
import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
Review comment:
should go straight after line 45 - no gap
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
Review comment:
we leave out all of test/resources to stop log4j files, site configs etc getting onto the classpath of apps downstream -so making it impossible for them to choose their own options
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
+ conf.setInt("yarn.scheduler.capacity.root.default.capacity", 100);
+ yarnCluster = new MiniYARNCluster(getClass().getName(), 1, 1, 1, 1);
+ yarnCluster.init(conf);
+ yarnCluster.start();
}
@After
public void clusterDown() {
if (cluster != null) {
cluster.close();
}
+ if (yarnCluster != null) {
Review comment:
all services are closeable, so use `IOUtil.cleanupWithLogger()` & let it handle null checks and exceptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-716738274
Thanks. I added a test for this. Note as mentioned previously, this will fail currently because of HADOOP-17327.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r520092586
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
+ conf.setInt("yarn.scheduler.capacity.root.default.capacity", 100);
+ yarnCluster = new MiniYARNCluster(getClass().getName(), 1, 1, 1, 1);
+ yarnCluster.init(conf);
+ yarnCluster.start();
}
@After
public void clusterDown() {
if (cluster != null) {
cluster.close();
}
+ if (yarnCluster != null) {
Review comment:
Nice. Didn't know this util method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-715645807
I'm also looking at shading issues at the same time. Mvn verify is broken in trunk otherwise I would ask you to run...
I'm not sure I understand the use case here. Is the MiniYARNCluster simply broken out of box? Or that Spark wants to test using MiniYARNCluster but substitue a custom version of BouncyCastle?
Our client jar contains only one IT ITUseMiniCluster and it checks only hdfs and webhdfs clients. How about adding an IT for MiniYARNCluster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-715639030
cc @steveloughran @jojochuang @aajisaka - could you please take a look? thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-716541272
makes sense, especially given bouncy-castle's export nature...it's a special module.
A test will defend against regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-716788701
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 43s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 23s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 0m 26s | | branch/hadoop-client-modules/hadoop-client-api no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-runtime no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 22s | | branch/hadoop-client-modules/hadoop-client-minicluster no findbugs output file (findbugsXml.xml) |
| +0 :ok: | findbugs | 0m 23s | | branch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (findbugsXml.xml) |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| -0 :warning: | checkstyle | 0m 14s | [/diff-checkstyle-hadoop-client-modules.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/artifact/out/diff-checkstyle-hadoop-client-modules.txt) | hadoop-client-modules: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML file. |
| -1 :x: | shadedclient | 14m 42s | | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-api has no data from findbugs |
| +0 :ok: | findbugs | 0m 20s | | hadoop-client-modules/hadoop-client-runtime has no data from findbugs |
| +0 :ok: | findbugs | 0m 19s | | hadoop-client-modules/hadoop-client-minicluster has no data from findbugs |
| +0 :ok: | findbugs | 0m 18s | | hadoop-client-modules/hadoop-client-integration-tests has no data from findbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-integration-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 97m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux bfc9aec29534 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 872440610f0 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/testReport/ |
| Max. process+thread count | 402 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-integration-tests U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-725668728
@steveloughran you mean in https://dist.apache.org/repos/dist/release/hadoop/common/KEYS ? no I haven't done that yet. Thanks for the info, do you know any doc to walk through this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-715635235
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | mvnsite | 1m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 19s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 55s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 84m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml |
| uname | Linux 5a2457d3a0b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74634eb0027 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r512175536
##########
File path: hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
Review comment:
another issue: we currently don't bundle YARN test resources like capacity-scheduler.xml in hadoop-client-minicluster so this is a temporary workaround. I'll open another JIRA for this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-724833749
Thanks @steveloughran ! yes please backport it to branch-3.3 as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2411: HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2411:
URL: https://github.com/apache/hadoop/pull/2411#issuecomment-715635235
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | mvnsite | 1m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 19s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 55s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-api in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-runtime in the patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-minicluster in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 84m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml |
| uname | Linux 5a2457d3a0b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74634eb0027 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2411/1/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org