You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/06/22 19:28:30 UTC
[GitHub] [ozone] adoroszlai opened a new pull request #2356: HDDS-3292. Support Hadoop 3.3
adoroszlai opened a new pull request #2356:
URL: https://github.com/apache/ozone/pull/2356
## What changes were proposed in this pull request?
Upgrade Ozone's Hadoop dependency to 3.3.1. The biggest change required is upgrading Mockito (due to HADOOP-14178).
This does not yet add MapReduce acceptance test with the new version, as that requires new docker images first.
https://issues.apache.org/jira/browse/HDDS-3292
## How was this patch tested?
Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/959385474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #2356:
URL: https://github.com/apache/ozone/pull/2356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r656696838
##########
File path: hadoop-ozone/dist/src/main/smoketest/spnego/web.robot
##########
@@ -41,7 +41,7 @@ Verify SPNEGO enabled URL
[arguments] ${url}
Run Keyword if '${SECURITY_ENABLED}' == 'true' Execute kdestroy
${result} = Execute curl --negotiate -u : -v -s -I ${url}
- Should contain ${result} 401 Unauthorized
+ Should contain ${result} 401 Authentication required
Review comment:
This is changed by the Jetty version upgrade in Hadoop 3.3.1 (HADOOP-17371 ) where Hadoop adds back the ability to specify a custom error message (the standard error message is "401 Unauthorized") for Jetty >= 9.4.21.
By the HTTP standard, applications are not supposed to depend on the error message (only error code) so I think this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r656698633
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
BTW, discussed with @smengcl offline, it seems strange we load hdfs-client jar (which is the culprit of this test error). Do we really need hdfs-client in Ozone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ayushtkn commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r657898861
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
>What I found strange was that other tests with setQuietMode(false) are passing.
There is no FileSystem.get(conf) in those classes I think. The problem surfaces because of that only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r666594427
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
@adoroszlai Ok. That works for me. Thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r656814907
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
> it seems strange we load hdfs-client jar (which is the culprit of this test error)
OM uses `URLConnectionFactory` from this lib.
What I found strange was that other tests with `setQuietMode(false)` are passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r665142695
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
@smengcl I don't understand the importance of keeping quiet mode disabled. The config file is not really required as everything works in the default quiet mode. What is the benefit of adding the empty config file?
I have merged the change to unblock follow-up work on protobuf upgrade. If you have any concerns about the change to this single test, please file a new issue and we can discuss/address it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r656699849
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
@adoroszlai I thought we were intentionally placing `conf.setQuietMode(false)` here uncover any warnings in the tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a change in pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#discussion_r656698062
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
##########
@@ -139,7 +139,6 @@ public static void init() throws Exception {
.setOMServiceId(omServiceId)
.setNumOfOzoneManagers(numOfOMs)
.build();
- conf.setQuietMode(false);
Review comment:
the quiet mode was suppressed because Hadoop 3.3 expects hdfs-rbf-default.xml which we don't have.
Another workaround proposed by @smengcl is to add a dummy hdfs-rbf-default.xml as a resource file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2356: HDDS-3292. Support Hadoop 3.3
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2356:
URL: https://github.com/apache/ozone/pull/2356#issuecomment-875381921
Thanks @ayushtkn, @jojochuang, @mukul1987 and @smengcl for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org