You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/04/09 14:05:25 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1948:
HADOOP-16855. wildfly classpath/loading issues
steveloughran opened a new pull request #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948
Contributed by Steve Loughran
Successor to HADOOP-16346.
* completely remove wildfly jar as a dependency of hadoop-aws
* socket factory to catch and swallow all
* and verify that in tests
* tests which also verify that in the default factory mode,
classloader errors are caught and downgrade to jvm
* and in openssl mode, the s3a binding explicitly catches
these failures and handles too
This means
s3a:
when widlfy doesn't load for classpath or wildfly/openssl issues, S3A will
downgrade to jvm, irrespective of SSL mode option.
abfs:
when widlfy doesn't load for classpath or wildfly/openssl issues,
default: swallow and downgrade to jvm
openssl: exception thrown; you can't talk to azure.
Yes, these are different behaviours. But abfs has always mandated the wildfly
library S3A has never done this and starting now will only surprise people. If
people/project downstream want to use it -fine. If they are running on the
machines where open SSL is found and is compatible -they will actually see a
speed up. Without that, it is utterly superfluous and simply any other source
of class path dependency and stack trace issues.
Change-Id: I74c4b8a1876b5c28d7d96fe3a6dc8e22ffbdc1f7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sahilTakiar commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
sahilTakiar commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-612185456
hmm I guess the only third-party dependency that hadoop-aws has is the aws sdk, so I can see the hesitation in adding another dependency. on the other hand, the wildfly jar is just a runtime dependency, not compile time.
the main concern I have is with the behavior change to the `openssl` option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims edited a comment on issue #1948: HADOOP-16986.
s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-614001981
I tested the patch on CentOS 8 and openssl-1.1.1-8.el8.x86_64. I got expected behaviour.
Without wildfly, `fs.s3a.ssl.channel.mode=openssl` raised NoClassDefFoundError and `fs.s3a.ssl.channel.mode=default` falled back to JSSE default.
With wildfly-openssl-1.0.10.Final in the classpath, it is used by supplying `HADOOP_OPTS="-Dorg.wildfly.openssl.path=/lib64"`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-614059053
@iwasakims -nice to see tokyo tested; been playing with stockholm today after some reports of trouble with our SDK version, which predates the region.
> should this be in 3.3.0?
I'm thinking: yes, otherwise we will add a new downstream dependency. Which should go into the release notes.
I've been playing with this a little bit more, based on Sahil's comments about env vars.
Once you take wildfly off the compile CP of hadoop-aws, it isn't picked up on the command line in ~/.hadoop-env
```
HADOOP_OPTIONAL_TOOLS="hadoop-aws"
```
and the "hadoop s3guard" command doesn't find it either
so unless you get wildfly into your CP in some other way, you can't use the pure openssl binding from the CLI. "default" is fine, just not "openssl"
What to do? I like the resilience we've added here, but worry about whether its now overreacting
Maybe we should
1. reinstate it as a compile dependency (i.e revert the pom change here); fix the new tests to cope with the fact it's found (i.e failure fallback isn't going to be tested)
2. declare that it should be on the CP (release notes), but that it isn't mandatory
3. hadoop-cloud-storage has been pulling it in via azure anyway -maybe we should add an explicit declaration too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-615297603
last test run (-s3guard, auth scale) with s3 ireland -proxy config tests filed (JIRA issued), and we got a failure of a job commit in an MR job. Got more logs there and hope to make progress next week [HADOOP-16798](https://issues.apache.org/jira/browse/HADOOP-16798)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-611547679
Sorry, I hadn't realised the previous patch was going to require wildfly
everywhere; I wouldn't make sure that was an otherwise.
I intend to get this patch into branch 3.3 as well, so only 3.3.0 has the
extra dependency.
Compared to #1929, this patch doesn't move wildfly dependency up from runtime to compile -this should not be needed. I've verified it gets through the tools-list and hadoop-cloud-storage dependencies, and am building a full release to see what goes on there.
then I'll do the azure and s3a test runs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on issue #1948: HADOOP-16986. s3a to
not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-614001981
I tested the patch on CentOS 8 and openssl-1.1.1-8.el8.x86_64. I got expected behaviour.
Without wildfly, `fs.s3a.ssl.channel.mode=openssl` raised NoClassDefFoundError and `fs.s3a.ssl.channel.mode=openssl` falled back to JSSE default.
With wildfly-openssl-1.0.10.Final in the classpath, it is used by supplying `HADOOP_OPTS="-Dorg.wildfly.openssl.path=/lib64"`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-611613061
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 19s | trunk passed |
| +1 :green_heart: | compile | 17m 9s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 12s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 16m 17s | the patch passed |
| +1 :green_heart: | javac | 16m 17s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 37s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 20s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed |
| +1 :green_heart: | findbugs | 3m 30s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 34s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. |
| | | 123m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 0adcb8aa92c1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c04a484 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/testReport/ |
| Max. process+thread count | 2557 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1948:
HADOOP-16986. s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-611613061
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 19s | trunk passed |
| +1 :green_heart: | compile | 17m 9s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 12s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 16m 17s | the patch passed |
| +1 :green_heart: | javac | 16m 17s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 37s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 20s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed |
| +1 :green_heart: | findbugs | 3m 30s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 34s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. |
| | | 123m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 0adcb8aa92c1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c04a484 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/testReport/ |
| Max. process+thread count | 2557 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran edited a comment on issue #1948:
HADOOP-16986. s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613920202
* checkstyle on length of comments and debug logs; fixed.
* fixed site issues, also did another review of the openssl section
not done a test rerun except the new test suite from the IDE, one where I haven't yet done a mvn import to pick up the changes in classpaths of s3a.
Notable that on the mac openssl does seem to be found in this situation.
```
org.junit.ComparisonFailure: [Channel mode of socket factory created with mode Default]
Expected :Default_JSSE
Actual :OpenSSL
at org.apache.hadoop.fs.s3a.TestOpenSSLClasspaths.expectBound(TestOpenSSLClasspaths.java:111)
at org.apache.hadoop.fs.s3a.TestOpenSSLClasspaths.testDefaultDowngrades(TestOpenSSLClasspaths.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613915399
tested: s3 london with `-Dparallel-tests -DtestsThreadCount=6 -Dscale -Dmarkers=delete -Ds3guard -Ddynamo` and fs.s3a.ssl.channel.mode = default. This was macos, no openSSL AFAIK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1948:
HADOOP-16855. wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#discussion_r408003485
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/DelegatingSSLSocketFactory.java
##########
@@ -149,11 +149,11 @@ private void initializeSSLContext(SSLChannelMode preferredChannelMode)
throws NoSuchAlgorithmException, KeyManagementException {
switch (preferredChannelMode) {
case Default:
- if (!openSSLProviderRegistered) {
- OpenSSLProvider.register();
- openSSLProviderRegistered = true;
- }
try {
+ if (!openSSLProviderRegistered) {
Review comment:
I'd put it here so that if someone asked for openSSL and there were problems, it really would fail. It was in default where wildfly load (and pointer errors) triggered catch and downgrade.
For the S3A Binding there already was some classpath catching, so I Added it there too so that any failure here just falls back to the default AWS SDK binding. For abfs I've left it alone
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1948: /HADOOP-16986. wildfly
classpath/loading issues
Posted by GitBox <gi...@apache.org>.
bgaborg commented on issue #1948: /HADOOP-16986. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613604776
LGTM, should we mention this somewhere in the docs? maybe troubleshooting?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-615311403
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 10s | trunk passed |
| +1 :green_heart: | compile | 16m 53s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 36s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 20s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 23s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 46s | trunk passed |
| +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | trunk passed |
| -0 :warning: | patch | 1m 34s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | the patch passed |
| +1 :green_heart: | compile | 17m 12s | the patch passed |
| +1 :green_heart: | javac | 17m 12s | the patch passed |
| -0 :warning: | checkstyle | 2m 35s | root: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 2m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 20s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | the patch passed |
| +1 :green_heart: | findbugs | 3m 31s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 8m 38s | hadoop-common in the patch failed. |
| -1 :x: | unit | 0m 53s | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 37s | ASF License check generated no output? |
| | | 121m 45s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.ha.TestZKFailoverControllerStress |
| | hadoop.crypto.key.TestValueQueue |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux afeae74f7b15 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 2fe122e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/testReport/ |
| Max. process+thread count | 2837 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613606566
hmm. docs. yeah. you're probably right.
renamed this patch to a new JIRA as I've merged @bilaharith's original patch in to branch-3.3+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1948:
HADOOP-16855. wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#discussion_r407653128
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/DelegatingSSLSocketFactory.java
##########
@@ -149,11 +149,11 @@ private void initializeSSLContext(SSLChannelMode preferredChannelMode)
throws NoSuchAlgorithmException, KeyManagementException {
switch (preferredChannelMode) {
case Default:
- if (!openSSLProviderRegistered) {
- OpenSSLProvider.register();
- openSSLProviderRegistered = true;
- }
try {
+ if (!openSSLProviderRegistered) {
Review comment:
Wouldn't it be better to move the code within try-catch into a new method so that the same can be resued for the case openSSL as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613665288
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 5s | trunk passed |
| +1 :green_heart: | compile | 17m 1s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 38s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 21s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 46s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 13s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 16m 21s | the patch passed |
| +1 :green_heart: | javac | 16m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 35s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 11s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed |
| +1 :green_heart: | findbugs | 3m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 6s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| -1 :x: | asflicense | 0m 54s | The patch generated 25 ASF License warnings. |
| | | 121m 54s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| | hadoop.util.TestReflectionUtils |
| | hadoop.metrics2.source.TestJvmMetrics |
| | hadoop.conf.TestConfiguration |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1a269155ddf7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0ad0102 |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1377 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1948:
HADOOP-16986. s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#discussion_r408691832
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
##########
@@ -539,7 +539,7 @@ in Java 9, so if `default_jsse` is specified and applications run on Java
includes GCM in the list of cipher suites on Java 8, so it is equivalent to
running with the vanilla JSSE.
-### OpenSSL Acceleration
+### <a name="openssl"></> OpenSSL Acceleration
Review comment:
aah. thx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-614000250
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 59s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 54s | trunk passed |
| +1 :green_heart: | compile | 22m 43s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 24m 28s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | trunk passed |
| +0 :ok: | spotbugs | 1m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 54s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 48s | the patch passed |
| +1 :green_heart: | compile | 21m 42s | the patch passed |
| +1 :green_heart: | javac | 21m 42s | the patch passed |
| +1 :green_heart: | checkstyle | 3m 31s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 18m 7s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 46s | the patch passed |
| +1 :green_heart: | findbugs | 4m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 39s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 45s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 181m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint |
| uname | Linux e3b302aedf2a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 94f7470 |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/4/testReport/ |
| Max. process+thread count | 2810 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613673982
The latest patch does say "if you ask for openssl, enjoy the stack trace"; docs explain why you get the stack and actions to take.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613339093
> Sure, wildfly is another dependency, but it has no transitive dependencies, so not sure what the harm is with including it. It doesn't do any weird shading of other dependencies either. This also forces
1 .It's another dependency that everyone downstream has to get right
2. It's a new one which nobody was expecting and if they don't get their packaging done it will be missing: S3A stops working.
3. if it is on your CP and there's a native OpenSSL implementation which wildfly was incompatible with, you get to see a stack trace
4. and if someone somehow gets a 1.0.4.Final version on the CP (and remember, it's been hidden in third-partly JARs like the azure-datalake JAR), then instantiating S3 or ABFS clients will crash on any Linux distro with 1.1.1 in.
issues 3 & 4 is critical for me, even on the "this is harmless" execution path, wildfly on the CP could break things. Yes, that's probably a bug in wildfly, and it's fixed now. but that doesn't mean that a new openSSL release will cause it to come back.
* there are lots of deployments where wildfly is not only not needed, if it causes things to break it is in fact dangerous(
bq. the main concern I have is with the behavior change to the openssl option.)
An ASF Hadoop release with wildfly support hasn't shipped. There is no behaviour change. Yes, there may be changes with releases by other entities, but as past changes have long determined, that cannot be a reason to hole back changes in the hadoop codebase.
bq. I can see the hesitation in adding another dependency. on the other hand, the wildfly jar is just a runtime dependency, not compile time.
If you need it on the classpath downstream the difference between compile and runtime is moot. If it is not there, your code doesn't work. If it is there and incompatible with your next distribution, your application breaks. This is not a transparent change.
Put differently: If I had noticed the previous patch mandated widfly as a dependency, it would have been held back for an update to remove that requirement.
> the main concern I have is with the behavior change to the openssl option.
The S3A Binding already handled lots of failures including class not found. I'm just expanding it a bit, with the goal of resilience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613718084
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 17s | trunk passed |
| +1 :green_heart: | compile | 16m 58s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 36s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 19s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | trunk passed |
| +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 12s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | the patch passed |
| +1 :green_heart: | compile | 16m 16s | the patch passed |
| +1 :green_heart: | javac | 16m 16s | the patch passed |
| -0 :warning: | checkstyle | 2m 38s | root: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| -1 :x: | mvnsite | 0m 51s | hadoop-aws in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 46s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | the patch passed |
| +1 :green_heart: | findbugs | 3m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 31s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 37s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 125m 11s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint |
| uname | Linux 440915eac474 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0ad0102 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/3/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/3/testReport/ |
| Max. process+thread count | 1377 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613339740
Test results: all good, the usual S3A versioned bucket test failure for which I've a patch outstanding; abfs happy *and fast* (Trivia: maven test launch doesn't work with AWS corretto JDK8 distro)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16986. s3a
to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613920202
* checkstyle on length of comments and debug logs; fixed.
* fixed site issues, also did another review of the openssl section
not done a test rerun except the new test suite from the IDE, one where I haven't yet done a mvn import to pick up the changes in classpaths of s3a.
Notable that on the mac openssl does seem to be found in this situation.
```
org.junit.ComparisonFailure: [Channel mode of socket factory created with mode Default]
Expected :Default_JSSE
Actual :OpenSSL
<Click to see difference>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at org.apache.hadoop.fs.s3a.TestOpenSSLClasspaths.expectBound(TestOpenSSLClasspaths.java:111)
at org.apache.hadoop.fs.s3a.TestOpenSSLClasspaths.testDefaultDowngrades(TestOpenSSLClasspaths.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1948:
HADOOP-16986. s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613665288
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 5s | trunk passed |
| +1 :green_heart: | compile | 17m 1s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 38s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 21s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 46s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 13s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 16m 21s | the patch passed |
| +1 :green_heart: | javac | 16m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 35s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 11s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed |
| +1 :green_heart: | findbugs | 3m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 6s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| -1 :x: | asflicense | 0m 54s | The patch generated 25 ASF License warnings. |
| | | 121m 54s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| | hadoop.util.TestReflectionUtils |
| | hadoop.metrics2.source.TestJvmMetrics |
| | hadoop.conf.TestConfiguration |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1948 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1a269155ddf7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0ad0102 |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 1377 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1948/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sahilTakiar commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
sahilTakiar commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-612146200
I don't agree with this approach.
> and in openssl mode, the s3a binding explicitly catches
> these failures and handles too
Why? The whole point of openssl mode is that if you enable it, you want the code to fail if openssl cannot be setup. The point of default mode is that it should be best effort.
> Without that, it is utterly superfluous and simply any other source
> of class path dependency and stack trace issues.
Not sure I understand the concern with the classpath issues. Sure, wildfly is another dependency, but it has no transitive dependencies, so not sure what the harm is with including it. It doesn't do any weird shading of other dependencies either. This also forces users to pick which version of wildfly to use, rather than S3A saying we support version 'x.y.z'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on issue #1948: HADOOP-16986. s3a to
not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
iwasakims commented on issue #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-614021477
LGTM. TestOpenSSLClasspaths passed. I got no relevant failure on `mvn verify -T 1C -Dparallel-tests -Ds3guard -Ddynamo` against Tokyo region. (on CentOS 8, openssl-1.1.1-8.el8.x86_64, java-1.8.0-openjdk-devel-1.8.0.232.b09-0.el8_0.x86_64).
Should this be in 3.3.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on a change in pull request #1948:
HADOOP-16986. s3a to not need wildfly on the classpath
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #1948: HADOOP-16986. s3a to not need wildfly on the classpath
URL: https://github.com/apache/hadoop/pull/1948#discussion_r408578893
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
##########
@@ -539,7 +539,7 @@ in Java 9, so if `default_jsse` is specified and applications run on Java
includes GCM in the list of cipher suites on Java 8, so it is equivalent to
running with the vanilla JSSE.
-### OpenSSL Acceleration
+### <a name="openssl"></> OpenSSL Acceleration
Review comment:
`mvn site` fails due to invalid tag `</>`. should be `</a>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1948: HADOOP-16855.
wildfly classpath/loading issues
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1948: HADOOP-16855. wildfly classpath/loading issues
URL: https://github.com/apache/hadoop/pull/1948#issuecomment-613400074
+email from Sahil says that it only gets through the HADOOP_OPTIONAL_TOOLS env var if its listed as compile time; somehow the bash stitching together needs it.
So @bilaharith's change does need to come in here too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org