You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ferdelyi (via GitHub)" <gi...@apache.org> on 2023/05/10 13:35:55 UTC
[GitHub] [hadoop] ferdelyi opened a new pull request, #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
ferdelyi opened a new pull request, #5638:
URL: https://github.com/apache/hadoop/pull/5638
… SSL/TLS into the common library
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
In order to enable SSL/TLS for the ZK Client to communicate with the secured ZK Server, the configuration parameters for the truststore and the keystore are introduced into the core-default.xml.
These parameters then used in the ZKCuratorManager class to set up the ZK Client for SSL/TLS encryption when the caller sets the sslEnabled flag true.
The secured ZK Client uses Netty, hence the dependency is included in the pom.xml.
A separate unit test class was introduced (TestSecureZKCuratorManager), as the secured server needs a very different setup from the already implemented unit test (TestZKCuratorManager).
### How was this patch tested?
This patch was tested with a unit test.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193971258
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
+ throws ConfigurationException {
+ LOG.info("Configuring the ZooKeeper client to use SSL/TLS encryption for connecting to the ZooKeeper server.");
+ if (StringUtils.isEmpty(this.keystoreLocation)) {
Review Comment:
Created for this purpose the validateSslConfiguration method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1204408155
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -34,31 +39,32 @@
import org.apache.hadoop.fs.CommonConfigurationKeys;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.authentication.util.JaasConfiguration;
+import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.util.ZKUtil;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Stat;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hadoop.util.Preconditions;
/**
* Helper class that provides utility methods specific to ZK operations.
*/
@InterfaceAudience.Private
public final class ZKCuratorManager {
- private static final Logger LOG =
- LoggerFactory.getLogger(ZKCuratorManager.class);
+ private static final Logger LOG = LoggerFactory.getLogger(ZKCuratorManager.class);
Review Comment:
I like cleaning this kind of things but it adds too much churn for a change that should be more contained.
I would prefer having a PR just for cleaning up the code than adding some feature while fixing other things in the side.
Let's avoid in this PR:
* Changing import.
* Adjusting line length breaks.
* Adjusting comments.
* Changing order.
Happy to review the followup for those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1563135627
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 26m 19s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 15m 36s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 27s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| -1 :x: | compile | 6m 50s | [/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 50s | [/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 6m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 46s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 12s | | the patch passed |
| -1 :x: | shadedclient | 8m 54s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 36s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 136m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 029a5a240adf 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f7c605e27bf74811e02eb4f1585364ee0c757f44 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/testReport/ |
| Max. process+thread count | 3105 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/25/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199521669
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
Review Comment:
I see. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1549882573
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 33s | | trunk passed |
| +1 :green_heart: | compile | 16m 16s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 25s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 16s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| -1 :x: | compile | 6m 50s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 50s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 23s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 23s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 53s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 10 new + 70 unchanged - 0 fixed = 80 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 16s | | the patch passed |
| -1 :x: | shadedclient | 9m 16s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 5s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 37s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 144m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux fdfff818430e 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33854b748fe38d4d5843f0702b40b268a811cd67 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/testReport/ |
| Max. process+thread count | 2906 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1548089533
Thank you Szilard for the CR.
The change was exclusively tested with the unit test, which is a kind of integration test, as a ZK Server was brought up and the communication tested between the client and the server.
This code change is in the common code base and there is no component calling it yet. Once YARN-11468 [Zookeeper SSL/TLS support] is implemented, we can test it in a real cluster environment.
Wondering if we should update the https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs-rbf/dependency-analysis.html page with the Netty dependency? The parameter descriptions are added to the commit to the core-default.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193988798
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
Review Comment:
Taken the skeleton of the class from the TestZKCuratorManager.java and it was defined this way. Might it cause issues when multiple unit tests are running one after the other with the same server but with different configurations?
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory1 =
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1555369000
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 10s | | trunk passed |
| +1 :green_heart: | compile | 15m 35s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 30s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 15m 0s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 15m 0s | | the patch passed |
| +1 :green_heart: | compile | 14m 30s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 14m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 9s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 0s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/13/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 181m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 13421f96a9da 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 521931a58471fe5da6a2fd792f7550f5b737ef46 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/13/testReport/ |
| Max. process+thread count | 1332 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199521345
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
Review Comment:
There was also a missing one before method: validateSslConfiguration
Can you fix the javadoc as it is not starting with /** (but with: /*) ? Something must be odd with your formatter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560937862
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560135150
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 14s | | trunk passed |
| +1 :green_heart: | compile | 16m 15s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 56s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 16m 38s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 38s | | the patch passed |
| +1 :green_heart: | compile | 16m 32s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 22s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 5s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/19/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 194m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 0e55918adc25 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/19/testReport/ |
| Max. process+thread count | 1288 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1563142275
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 18s | | trunk passed |
| +1 :green_heart: | compile | 15m 42s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 19s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| -1 :x: | compile | 6m 38s | [/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 38s | [/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 12s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 6m 12s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 15s | | the patch passed |
| -1 :x: | shadedclient | 8m 53s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 48s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 36s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 141m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 4916f406b78d 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f7c605e27bf74811e02eb4f1585364ee0c757f44 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/testReport/ |
| Max. process+thread count | 1556 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1548259594
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 33s | | trunk passed |
| +1 :green_heart: | compile | 15m 37s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 7s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| -1 :x: | compile | 6m 40s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 40s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 12s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 12s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 45s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 13 new + 70 unchanged - 0 fixed = 83 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 16s | | the patch passed |
| -1 :x: | shadedclient | 9m 24s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 19m 7s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 33s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 146m 18s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.util.curator.TestZKCuratorManager |
| | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 472ddc358a21 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bdd0119a930fd4714fc5ad8c611297f27e8295a5 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/testReport/ |
| Max. process+thread count | 3150 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1191770472
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -157,12 +175,44 @@ public void start(List<AuthInfo> authInfos) throws IOException {
authInfos.add(new AuthInfo(zkAuth.getScheme(), zkAuth.getAuth()));
}
+ /* Pre-check on SSL/TLS client connection requirements to emit the name of the
+ configuration missing. It improves supportability. */
+ if(sslEnabled) {
+ if (StringUtils.isEmpty(conf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION))) {
+ throw new ConfigurationException(
Review Comment:
Nit: Can you extract this to a helper method?
Only the config name should be passed, the rest of the method body (even the exception message) can be the same.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
Review Comment:
Nit: Line break between methods.
Also it seems your formatter seems to be off, as method parentheses has an additional space before it after the parameters.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
+ throws ConfigurationException {
+ LOG.info("Configuring the ZooKeeper client to use SSL/TLS encryption for connecting to the ZooKeeper server.");
+ if (StringUtils.isEmpty(this.keystoreLocation)) {
Review Comment:
Nit: Here you could also extract the isempty check + throwing the exception to a method.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
Review Comment:
can be static final
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
+ throws ConfigurationException {
+ LOG.info("Configuring the ZooKeeper client to use SSL/TLS encryption for connecting to the ZooKeeper server.");
+ if (StringUtils.isEmpty(this.keystoreLocation)) {
+ throw new ConfigurationException(
+ "The keystore location parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.keystorePassword)) {
+ throw new ConfigurationException(
+ "The keystore password parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.truststoreLocation)) {
+ throw new ConfigurationException(
+ "The truststore location parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.truststorePassword)) {
+ throw new ConfigurationException(
+ "The truststore password parameter is empty for the ZooKeeper client connection.");
+ }
+ LOG.debug("Configuring the ZooKeeper client with {} {} location.",
+ this.keystoreLocation, CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION);
Review Comment:
maybe it's better / more straightforward as:
```suggestion
LOG.debug("Configuring the ZooKeeper client with {} location: {}",
this.keystoreLocation, CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION);
```
##########
hadoop-common-project/hadoop-common/pom.xml:
##########
@@ -342,6 +342,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
Review Comment:
What's the reason of adding the netty dependencies here?
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
Review Comment:
can be static final
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
Review Comment:
For better readability, these could be extracted to static final fields or local variables.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
Review Comment:
Can this be removed?
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
Review Comment:
Is there a meaning of this number?
Can it be calculated and saved to a static final int constant?
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
Review Comment:
what's the reason of setting it to null?
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory1 =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD));
+ ZooKeeper zk1 = factory1.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk1);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ ClientX509Util x509Util = new ClientX509Util();
Review Comment:
IntelliJ gives me a warning: 'ClientX509Util' used without 'try'-with-resources statement
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory1 =
Review Comment:
nit: Can you call the variables "factory" and "zk"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193895248
##########
hadoop-common-project/hadoop-common/pom.xml:
##########
@@ -342,6 +342,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
Review Comment:
The secured ZK Client uses Netty, hence the dependency is included in the pom.xml. Added netty-handler and netty-transport-native-epoll dependency to the pom.xml based on ZOOKEEPER-3494 - "No need to depend on netty-all (SSL)". Added this info to the Jira.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1548167321
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 19s | | trunk passed |
| +1 :green_heart: | compile | 15m 39s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 15s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| -1 :x: | compile | 6m 38s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 38s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 7s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 7s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 48s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 13 new + 70 unchanged - 0 fixed = 83 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 13s | | the patch passed |
| -1 :x: | shadedclient | 8m 59s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 38s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 37s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 145m 40s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.util.curator.TestZKCuratorManager |
| | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux ac50c84cad75 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 45ed03020db65115b6b7f634f4199ed10053d83f |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/testReport/ |
| Max. process+thread count | 1327 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1551305126
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 12s | | trunk passed |
| +1 :green_heart: | compile | 17m 30s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 55s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 37s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 16m 50s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 50s | | the patch passed |
| +1 :green_heart: | compile | 15m 51s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 15m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 2s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 54s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/11/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 189m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux b2912cda63d1 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ae8e5f5af0aea960d8a4ba02785818a27bf3e348 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/11/testReport/ |
| Max. process+thread count | 3019 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560135690
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 36s | | trunk passed |
| +1 :green_heart: | compile | 17m 32s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 50s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 16m 47s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 47s | | the patch passed |
| +1 :green_heart: | compile | 15m 50s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 15m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 44s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 53s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/20/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 195m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 9fa60a5d8292 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/20/testReport/ |
| Max. process+thread count | 1239 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/20/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193971771
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
+ throws ConfigurationException {
+ LOG.info("Configuring the ZooKeeper client to use SSL/TLS encryption for connecting to the ZooKeeper server.");
+ if (StringUtils.isEmpty(this.keystoreLocation)) {
+ throw new ConfigurationException(
+ "The keystore location parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.keystorePassword)) {
+ throw new ConfigurationException(
+ "The keystore password parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.truststoreLocation)) {
+ throw new ConfigurationException(
+ "The truststore location parameter is empty for the ZooKeeper client connection.");
+ }
+ if (StringUtils.isEmpty(this.truststorePassword)) {
+ throw new ConfigurationException(
+ "The truststore password parameter is empty for the ZooKeeper client connection.");
+ }
+ LOG.debug("Configuring the ZooKeeper client with {} {} location.",
+ this.keystoreLocation, CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION);
Review Comment:
Thank you for the heads-up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1555433823
> Thank you Szilard for the CR.
>
> The change was exclusively tested with the unit test, which is a kind of integration test, as a ZK Server was brought up and the communication tested between the client and the server.
>
> This code change is in the common code base and there is no component calling it yet. Once [YARN-11468](https://issues.apache.org/jira/browse/YARN-11468) [Zookeeper SSL/TLS support] is implemented, we can test it in a real cluster environment.
>
> Wondering if we should update the [hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs-rbf/dependency-analysis.html](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs-rbf/dependency-analysis.html) page with the Netty dependency? The parameter descriptions are added to the commit to the core-default.xml.
I see, thanks for the info. Didn't know about the YARN jira.
I don't think you need to update the dependency report, TBH I never updated it and I don't know how it's generated. Probably copied from the output of some script? Our codebase might have a reference to this somewhere, in markdown files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1562493374
Filed bug report on Yetus: [INFRA-24640 ](https://issues.apache.org/jira/browse/INFRA-24640)- "Yetus fails ASF license check on deleted files"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199521345
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
Review Comment:
There was also a missing one before method: validateSslConfiguration
I fixed the javadoc as it was not starting with /** (but with: /*) before committing the change. Something must be odd with your formatter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1575670756
Thanks @ferdelyi for working on this.
Latest patch LGTM, committed to trunk.
Thaknks @goiri for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560121784
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 2s | | trunk passed |
| +1 :green_heart: | compile | 15m 36s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 22s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 16s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 15m 3s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 15m 3s | | the patch passed |
| +1 :green_heart: | compile | 14m 16s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 14m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 1s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 2s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/16/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 183m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux f01ffa2966cd 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/16/testReport/ |
| Max. process+thread count | 1403 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1542486038
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 29s | | trunk passed |
| +1 :green_heart: | compile | 20m 26s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 22m 18s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 48s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| -1 :x: | compile | 7m 30s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 7m 30s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 57s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 57s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 48s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 30 new + 70 unchanged - 0 fixed = 100 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 0m 47s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 19s | | the patch passed |
| -1 :x: | shadedclient | 9m 15s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 25s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 163m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux fea6cafe37f9 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 88c3b21c2148da020041fed99f7b4296e04803cc |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/testReport/ |
| Max. process+thread count | 1600 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193974492
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
Review Comment:
It can't be calculated, this value is tuned based on the use case. The encrypted channell has some additional payload, hence the default jute.maxbuffer is not satisfactory. Established this number by defining a larger number than 369296129, which was reported as an exception when running the unit test with default values. Modified it to 400000000, so it is easier on the eyes.
Let me save it to a static final int.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1554469170
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 35s | | trunk passed |
| +1 :green_heart: | compile | 15m 28s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 24s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 15m 1s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 15m 1s | | the patch passed |
| +1 :green_heart: | compile | 14m 22s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 14m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 4s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 3s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/12/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 174m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 663beea57d0a 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 521931a58471fe5da6a2fd792f7550f5b737ef46 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/12/testReport/ |
| Max. process+thread count | 1322 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199521447
##########
hadoop-common-project/hadoop-common/pom.xml:
##########
@@ -342,6 +342,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
Review Comment:
Cool, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193895248
##########
hadoop-common-project/hadoop-common/pom.xml:
##########
@@ -342,6 +342,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
Review Comment:
The secured ZK Client uses Netty, hence the dependency is included in the pom.xml. Added netty-handler and netty-transport-native-epoll dependency to the pom.xml based on ZOOKEEPER-3494 - "No need to depend on netty-all (SSL)". Adding this info to the Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1551021587
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 44s | | trunk passed |
| +1 :green_heart: | compile | 18m 32s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 48s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 44s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 36s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 20m 35s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 51s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 0s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/10/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 53s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 54s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/10/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 216m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux c345a53f825f 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / af50a75cedcdcdd54327620a9b77e4f4b6eb93ab |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/10/testReport/ |
| Max. process+thread count | 1253 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199527654
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -503,4 +644,50 @@ private void setJaasConfiguration(ZKClientConfig zkClientConfig) throws IOExcept
zkClientConfig.setProperty(ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, JAAS_CLIENT_ENTRY);
}
}
-}
\ No newline at end of file
+
+ /**
+ * Helper class to contain the Truststore/Keystore paths for the ZK client connection over
+ * SSL/TLS.
+ */
+ public static class TruststoreKeystore{
+ private static String keystoreLocation;
+ private static String keystorePassword;
+ private static String truststoreLocation;
+ private static String truststorePassword;
+ /** Configuration for the ZooKeeper connection when SSL/TLS is enabled.
+ * When a value is not configured, ensure that empty string is set instead of null.
+ * @param conf ZooKeeper Client configuration
+ */
+ public TruststoreKeystore(Configuration conf){
+
+ keystoreLocation =
+ StringUtils.defaultString(conf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION,
Review Comment:
Why the StringUtils.defaultString is needed?
I mean, conf.get() will return an empty string if the config is not found, given that you passed empty strings for all conf.get calls already.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
Review Comment:
Extracting these (at least 1, 100, and 10) to static finals would make this more readable and straightforward.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -452,21 +502,50 @@ public static class HadoopZookeeperFactory implements ZookeeperFactory {
private final String zkPrincipal;
private final String kerberosPrincipal;
private final String kerberosKeytab;
+ private final Boolean sslEnabled;
+ /**
+ * Constructor for the helper class to configure the ZooKeeper client connection.
+ * @param zkPrincipal Optional.
+ */
public HadoopZookeeperFactory(String zkPrincipal) {
this(zkPrincipal, null, null);
}
-
+ /**
+ * Constructor for the helper class to configure the ZooKeeper client connection.
+ * @param zkPrincipal Optional.
+ * @param kerberosPrincipal Optional. Use along with kerberosKeytab.
+ * @param kerberosKeytab Optional. Use along with kerberosPrincipal.
+ */
public HadoopZookeeperFactory(String zkPrincipal, String kerberosPrincipal,
String kerberosKeytab) {
+ this(zkPrincipal, kerberosPrincipal, kerberosKeytab, false,
+ new TruststoreKeystore(new Configuration())); }
+
+ /**
+ * Constructor for the helper class to configure the ZooKeeper client connection.
+ * @param zkPrincipal Optional.
+ * @param kerberosPrincipal Optional. Use along with kerberosKeytab.
+ * @param kerberosKeytab Optional. Use along with kerberosPrincipal.
+ * @param sslEnabled Flag to enable SSL/TLS ZK client connection for each component
+ * independently.
+ * @param truststoreKeystore TruststoreKeystore object containing the keystoreLocation,
+ * keystorePassword, truststoreLocation, truststorePassword for
+ * SSL/TLS connection when sslEnabled is set to true.
+ */
+ public HadoopZookeeperFactory(String zkPrincipal,
+ String kerberosPrincipal,
+ String kerberosKeytab,
+ boolean sslEnabled,
+ TruststoreKeystore truststoreKeystore) {
Review Comment:
the parameter called 'truststoreKeystore' is unused.
Please remove it
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
+ Configuration conf = new Configuration();
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server" +
+ ".NettyServerCnxnFactory");
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", JUTE_MAXBUFFER.toString());
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth" +
+ ".X509AuthenticationProvider");
+
+
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath +
+ "keystore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath +
+ "truststore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return conf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ new ZKCuratorManager.TruststoreKeystore(hadoopConf));
+ ZooKeeper zk = factory.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ try (ClientX509Util x509Util = new ClientX509Util()) {
+ //testing if custom values are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystorePasswdProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststorePasswdProperty()));
+ }
+ //testing if constant values hardcoded into the code are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", "true",
+ zk.getClientConfig().getProperty(ZKClientConfig.SECURE_CLIENT));
+ assertEquals("Validate that expected clientConfig is set in ZK config",
+ "org.apache.zookeeper.ClientCnxnSocketNetty",
+ zk.getClientConfig().getProperty(ZKClientConfig.ZOOKEEPER_CLIENT_CNXN_SOCKET));
+ }
+
+ @Test
+ public void testTruststoreKeystoreConfiguration(){
+ LOG.info("Entered to the testTruststoreKeystoreConfiguration test case.");
+ /**
Review Comment:
nit: Should be a block comment, now it's a dangling javadoc like this.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
+ Configuration conf = new Configuration();
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server" +
+ ".NettyServerCnxnFactory");
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", JUTE_MAXBUFFER.toString());
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth" +
+ ".X509AuthenticationProvider");
+
+
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath +
+ "keystore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath +
+ "truststore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return conf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ new ZKCuratorManager.TruststoreKeystore(hadoopConf));
+ ZooKeeper zk = factory.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ try (ClientX509Util x509Util = new ClientX509Util()) {
+ //testing if custom values are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystorePasswdProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststorePasswdProperty()));
+ }
+ //testing if constant values hardcoded into the code are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", "true",
+ zk.getClientConfig().getProperty(ZKClientConfig.SECURE_CLIENT));
+ assertEquals("Validate that expected clientConfig is set in ZK config",
+ "org.apache.zookeeper.ClientCnxnSocketNetty",
+ zk.getClientConfig().getProperty(ZKClientConfig.ZOOKEEPER_CLIENT_CNXN_SOCKET));
+ }
+
+ @Test
+ public void testTruststoreKeystoreConfiguration(){
+ LOG.info("Entered to the testTruststoreKeystoreConfiguration test case.");
+ /**
+ * By default the truststore/keystore configurations are not set, hence the values are null.
+ * Validate that the null values are converted into empty strings by the class.
+ */
+ Configuration conf = new Configuration();
+ new ZKCuratorManager.TruststoreKeystore(conf);
+
+ assertEquals("Validate that null value is converted to empty string.", "",
+ ZKCuratorManager.TruststoreKeystore.getKeystoreLocation());
+ assertEquals("Validate that null value is converted to empty string.", "",
+ ZKCuratorManager.TruststoreKeystore.getKeystorePassword());
+ assertEquals("Validate that null value is converted to empty string.", "",
+ ZKCuratorManager.TruststoreKeystore.getTruststoreLocation());
+ assertEquals("Validate that null value is converted to empty string.", "",
+ ZKCuratorManager.TruststoreKeystore.getTruststorePassword());
+
+ //Validate that non-null values will remain intact
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, "/keystore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "keystorePassword");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, "/truststore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "truststorePassword");
+ new ZKCuratorManager.TruststoreKeystore(conf);
Review Comment:
It doesn't make too much sense to invoke a constructor of TruststoreKeystore and store all values from the config as static fields.
Static is not really for this.
I would just convert all the 4 static fields to instance fields in TruststoreKeystore.
Funnily I only noticed this weirdness from the usage of this class in here in the testcase.
So basically, what will be changed here is to create a new instance of TruststoreKeystore with the conf and use the same getters, but on that object rather than in a static fashion like what you have currently.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
+ Configuration conf = new Configuration();
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server" +
+ ".NettyServerCnxnFactory");
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", JUTE_MAXBUFFER.toString());
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth" +
+ ".X509AuthenticationProvider");
+
+
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath +
+ "keystore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath +
+ "truststore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return conf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ new ZKCuratorManager.TruststoreKeystore(hadoopConf));
+ ZooKeeper zk = factory.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ try (ClientX509Util x509Util = new ClientX509Util()) {
+ //testing if custom values are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystorePasswdProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststorePasswdProperty()));
+ }
+ //testing if constant values hardcoded into the code are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", "true",
Review Comment:
Nit: "true" --> Boolean.TRUE.toString()
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
Review Comment:
could be primitive int.
Moreover, the name 'defaultValue' does not tell me anything.
Please add 2 static final ints and name them that resembles quorum port and election port.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
Review Comment:
Maybe setUpSecureConfig is a better name.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +557,72 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+
+ private void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util)
+ throws ConfigurationException {
+ validateSslConfiguration(
+ TruststoreKeystore.keystoreLocation,
+ TruststoreKeystore.keystorePassword,
+ TruststoreKeystore.truststoreLocation,
+ TruststoreKeystore.truststorePassword);
+ LOG.info("Configuring the ZooKeeper client to use SSL/TLS encryption for connecting to the " +
+ "ZooKeeper server.");
+ LOG.debug("Configuring the ZooKeeper client with {} location: {}.",
+ TruststoreKeystore.keystoreLocation,
+ CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION);
+ LOG.debug("Configuring the ZooKeeper client with {} location: {}.",
+ TruststoreKeystore.truststoreLocation,
+ CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION);
+
+ zkClientConfig.setProperty(ZKClientConfig.SECURE_CLIENT, "true");
+ zkClientConfig.setProperty(ZKClientConfig.ZOOKEEPER_CLIENT_CNXN_SOCKET,
+ "org.apache.zookeeper.ClientCnxnSocketNetty");
+ zkClientConfig.setProperty(x509Util.getSslKeystoreLocationProperty(),
+ TruststoreKeystore.keystoreLocation);
+ zkClientConfig.setProperty(x509Util.getSslKeystorePasswdProperty(),
+ TruststoreKeystore.keystorePassword);
+ zkClientConfig.setProperty(x509Util.getSslTruststoreLocationProperty(),
+ TruststoreKeystore.truststoreLocation);
+ zkClientConfig.setProperty(x509Util.getSslTruststorePasswdProperty(),
+ TruststoreKeystore.truststorePassword);
+ }
+
+ private void validateSslConfiguration(String keystoreLocation, String keystorePassword,
Review Comment:
All parameters are unused in this method. Please remove them.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
+ Configuration conf = new Configuration();
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server" +
Review Comment:
Nit: Getting the name of the class from the Class object of NettyServerCnxnFactory would be cleaner.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
+ Configuration conf = new Configuration();
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server" +
+ ".NettyServerCnxnFactory");
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", JUTE_MAXBUFFER.toString());
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth" +
+ ".X509AuthenticationProvider");
+
+
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath +
+ "keystore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath +
+ "truststore.jks");
+ conf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return conf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ new ZKCuratorManager.TruststoreKeystore(hadoopConf));
+ ZooKeeper zk = factory.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ try (ClientX509Util x509Util = new ClientX509Util()) {
+ //testing if custom values are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", keystorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslKeystorePasswdProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststoreLocation,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststoreLocationProperty()));
+ assertEquals("Validate that expected clientConfig is set in ZK config", truststorePassword,
+ zk.getClientConfig().getProperty(x509Util.getSslTruststorePasswdProperty()));
+ }
+ //testing if constant values hardcoded into the code are set properly
+ assertEquals("Validate that expected clientConfig is set in ZK config", "true",
+ zk.getClientConfig().getProperty(ZKClientConfig.SECURE_CLIENT));
+ assertEquals("Validate that expected clientConfig is set in ZK config",
+ "org.apache.zookeeper.ClientCnxnSocketNetty",
Review Comment:
Nit: Getting the name from the Class object will be cleaner.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client
+ * connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ static final Integer SECURE_CLIENT_PORT = 2281;
+ static final Integer JUTE_MAXBUFFER = 400000000;
+ static final File ZK_DATA_DIR = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ Integer defaultValue = -1;
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort", SECURE_CLIENT_PORT.toString());
+ customConfiguration.put("audit.enable", true);
+ this.hadoopConf = setUpSecure();
+ InstanceSpec spec = new InstanceSpec(ZK_DATA_DIR, SECURE_CLIENT_PORT,
+ defaultValue,
+ defaultValue,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ this.hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(this.hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public static Configuration setUpSecure() throws Exception {
+ return setUpSecure("src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/");
+ }
+ public static Configuration setUpSecure(String testDataPath) throws Exception {
Review Comment:
This method does not trow any Exception according to IntelliJ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1555435928
Hi @ferdelyi ,
I added a couple of review comments.
Could you please also add comments to the test class (e.g. in javadoc) about how the added certificate files, keystore and truststore files were generated, for example you may also add the commands that created those files.
As a reader of the test class, I woulnd't have any idea how those files got there and if any issue comes up in the future, the javadoc would tell. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1561676524
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 6s | | trunk passed |
| +1 :green_heart: | compile | 16m 41s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 3s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 16m 18s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 18s | | the patch passed |
| +1 :green_heart: | compile | 16m 9s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 0s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 1s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/23/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 185m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 6589edba2fa4 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bdbf03872942bcdd4e6ae48399653d02a29732c8 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/23/testReport/ |
| Max. process+thread count | 1900 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199521127
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -157,12 +175,44 @@ public void start(List<AuthInfo> authInfos) throws IOException {
authInfos.add(new AuthInfo(zkAuth.getScheme(), zkAuth.getAuth()));
}
+ /* Pre-check on SSL/TLS client connection requirements to emit the name of the
+ configuration missing. It improves supportability. */
+ if(sslEnabled) {
+ if (StringUtils.isEmpty(conf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION))) {
+ throw new ConfigurationException(
Review Comment:
I meant a method called something like validateX(String confKey) that throws the exception. The exception message is repeated 4 times, but it's not the end of the world if we don't do this. It's okay how it is now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1199522108
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
Review Comment:
As the setup is annotated with @Before, it always initializes this.server with an Object. I don't think setting it to null makes anything better. Let's leave it as it is, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1559894584
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 34s | | trunk passed |
| +1 :green_heart: | compile | 16m 13s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 23s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 15m 47s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 15m 47s | | the patch passed |
| +1 :green_heart: | compile | 14m 57s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 14m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/14/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 16s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 4s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/14/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 177m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 1a0085c52c62 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f841659394175924dec47eeec82335a13680d07 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/14/testReport/ |
| Max. process+thread count | 3159 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193948456
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -478,10 +558,53 @@ public ZooKeeper newZooKeeper(String connectString, int sessionTimeout,
if (zkClientConfig.isSaslClientEnabled() && !isJaasConfigurationSet(zkClientConfig)) {
setJaasConfiguration(zkClientConfig);
}
+ if (sslEnabled) {
+ setSslConfiguration(zkClientConfig);
+ }
return new ZooKeeper(connectString, sessionTimeout, watcher,
canBeReadOnly, zkClientConfig);
}
+ /**
+ * Configure ZooKeeper Client with SSL/TLS connection.
+ * @param zkClientConfig ZooKeeper Client configuration
+ * */
+ private void setSslConfiguration(ZKClientConfig zkClientConfig) throws ConfigurationException {
+ this.setSslConfiguration(zkClientConfig, new ClientX509Util());
+ }
+ public void setSslConfiguration(ZKClientConfig zkClientConfig, ClientX509Util x509Util )
Review Comment:
Thank you for the heads-up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1570164034
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 2s | | trunk passed |
| +1 :green_heart: | compile | 17m 39s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 1s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 16m 44s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 44s | | the patch passed |
| +1 :green_heart: | compile | 15m 57s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 15m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 1s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 19s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not generate ASF License warnings. |
| | | 191m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/29/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux e7c99d323ae3 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7e9ab3d63e82a21973371c48512833b0cc7a7827 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/29/testReport/ |
| Max. process+thread count | 3144 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/29/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth closed pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth closed pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
URL: https://github.com/apache/hadoop/pull/5638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1548472549
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 58s | | trunk passed |
| +1 :green_heart: | compile | 15m 34s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 24s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | compile | 0m 34s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 0m 34s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 0m 34s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 0m 34s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 45s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 10 new + 70 unchanged - 0 fixed = 80 total (was 70) |
| -1 :x: | mvnsite | 0m 55s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 0m 21s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | hadoop-common in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | spotbugs | 0m 21s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | shadedclient | 3m 0s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 21s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | asflicense | 0m 31s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 103m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux b9c78b452310 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 09296f5dc2506e2640b9eabb048f62b17803d51a |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/testReport/ |
| Max. process+thread count | 568 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1548286664
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 47s | | trunk passed |
| +1 :green_heart: | compile | 16m 11s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 15s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| -1 :x: | compile | 6m 55s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 6m 55s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 39s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 39s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 52s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 14 new + 70 unchanged - 0 fixed = 84 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 29s | | the patch passed |
| -1 :x: | shadedclient | 9m 16s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 18m 38s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 37s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 154m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.util.curator.TestZKCuratorManager |
| | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux e331b77452c0 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 559653cd1bb249e5b95fa3ab00c85a12dd61ad9e |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1549899596
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 48s | | trunk passed |
| +1 :green_heart: | compile | 16m 41s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 6s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| -1 :x: | compile | 7m 30s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 7m 30s | [/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 6m 25s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| -1 :x: | javac | 6m 25s | [/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | root in the patch failed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 50s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 10 new + 70 unchanged - 0 fixed = 80 total (was 70) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 27s | | the patch passed |
| -1 :x: | shadedclient | 9m 40s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 53s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 37s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 154m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux f1916b90d939 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33854b748fe38d4d5843f0702b40b268a811cd67 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/testReport/ |
| Max. process+thread count | 1291 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1200468351
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -503,4 +644,50 @@ private void setJaasConfiguration(ZKClientConfig zkClientConfig) throws IOExcept
zkClientConfig.setProperty(ZKClientConfig.LOGIN_CONTEXT_NAME_KEY, JAAS_CLIENT_ENTRY);
}
}
-}
\ No newline at end of file
+
+ /**
+ * Helper class to contain the Truststore/Keystore paths for the ZK client connection over
+ * SSL/TLS.
+ */
+ public static class TruststoreKeystore{
+ private static String keystoreLocation;
+ private static String keystorePassword;
+ private static String truststoreLocation;
+ private static String truststorePassword;
+ /** Configuration for the ZooKeeper connection when SSL/TLS is enabled.
+ * When a value is not configured, ensure that empty string is set instead of null.
+ * @param conf ZooKeeper Client configuration
+ */
+ public TruststoreKeystore(Configuration conf){
+
+ keystoreLocation =
+ StringUtils.defaultString(conf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION,
Review Comment:
I was getting some NPE exception based on my recollection without this, but can't reproduce it now, so removing it and will see during the build if it comes up again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1559905475
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 50s | | trunk passed |
| +1 :green_heart: | compile | 17m 1s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 57s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 16m 39s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 39s | | the patch passed |
| +1 :green_heart: | compile | 16m 19s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 41s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 4s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/15/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 55s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 58s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/15/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 186m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux fec8731db6de 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f841659394175924dec47eeec82335a13680d07 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/15/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193948150
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -157,12 +175,44 @@ public void start(List<AuthInfo> authInfos) throws IOException {
authInfos.add(new AuthInfo(zkAuth.getScheme(), zkAuth.getAuth()));
}
+ /* Pre-check on SSL/TLS client connection requirements to emit the name of the
+ configuration missing. It improves supportability. */
+ if(sslEnabled) {
+ if (StringUtils.isEmpty(conf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION))) {
+ throw new ConfigurationException(
Review Comment:
Extracted to the validateSslConfiguration() helper method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1193989267
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.test.InstanceSpec;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.FileContext.LOG;
+import static org.junit.Assert.assertEquals;
+
+
+/**
+ * Test the manager for ZooKeeper Curator when SSL/TLS is enabled for the ZK server-client connection.
+ */
+public class TestSecureZKCuratorManager {
+
+ private TestingServer server;
+ private ZKCuratorManager curator;
+ private Configuration hadoopConf;
+ private Integer secureClientPort = 2281;
+ private File zkDataDir = new File("testZkSSLClientConnectionDataDir");
+
+ @Before
+ public void setup() throws Exception {
+ //set zkServer
+ this.hadoopConf = setUpSecure();
+ Map<String, Object> customConfiguration = new HashMap<>();
+ customConfiguration.put("secureClientPort",this.secureClientPort.toString());
+ customConfiguration.put("audit.enable",true);
+
+ InstanceSpec spec = new InstanceSpec(
+ this.zkDataDir,
+ this.secureClientPort,
+ -1,
+ -1,
+ true,
+ 1,
+ 100,
+ 10,
+ customConfiguration);
+ this.server = new TestingServer(spec, true);
+ hadoopConf.set(CommonConfigurationKeys.ZK_ADDRESS, this.server.getConnectString());
+ this.curator = new ZKCuratorManager(hadoopConf);
+ this.curator.start(new ArrayList<>(), true);
+ }
+
+ public Configuration setUpSecure() throws Exception {
+ Configuration hadoopConf = new Configuration();
+ String testDataPath = "src/test/java/org/apache/hadoop/util/curator/resources/data";
+ System.setProperty("zookeeper.serverCnxnFactory", "org.apache.zookeeper.server.NettyServerCnxnFactory");
+ //System.setProperty("zookeeper.client.secure", "true");
+
+
+ System.setProperty("zookeeper.ssl.keyStore.location", testDataPath + "/ssl/keystore.jks");
+ System.setProperty("zookeeper.ssl.keyStore.password", "password");
+ System.setProperty("zookeeper.ssl.trustStore.location", testDataPath + "/ssl/truststore.jks");
+ System.setProperty("zookeeper.ssl.trustStore.password", "password");
+ System.setProperty("zookeeper.request.timeout", "12345");
+
+ System.setProperty("jute.maxbuffer", "469296129");
+
+ System.setProperty("javax.net.debug", "ssl");
+ System.setProperty("zookeeper.authProvider.x509", "org.apache.zookeeper.server.auth.X509AuthenticationProvider");
+
+
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION, testDataPath + "/ssl/keystore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD, "password");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION, testDataPath + "/ssl/truststore.jks");
+ hadoopConf.set(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD, "password");
+ return hadoopConf;
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.curator.close();
+ if (this.server != null) {
+ this.server.close();
+ this.server = null;
+ }
+ }
+
+ @Test
+ public void testSecureZKConfiguration() throws Exception {
+ LOG.info("Entered to the testSecureZKConfiguration test case.");
+ // Validate that HadoopZooKeeperFactory will set ZKConfig with given principals
+ ZKCuratorManager.HadoopZookeeperFactory factory1 =
+ new ZKCuratorManager.HadoopZookeeperFactory(
+ null,
+ null,
+ null,
+ true,
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD));
+ ZooKeeper zk1 = factory1.newZooKeeper(this.server.getConnectString(), 1000, null, false);
+ validateSSLConfiguration(
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_KEYSTORE_PASSWORD),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_LOCATION),
+ this.hadoopConf.get(CommonConfigurationKeys.ZK_SSL_TRUSTSTORE_PASSWORD),
+ zk1);
+ }
+
+ private void validateSSLConfiguration(
+ String keystoreLocation,
+ String keystorePassword,
+ String truststoreLocation,
+ String truststorePassword,
+ ZooKeeper zk){
+ ClientX509Util x509Util = new ClientX509Util();
Review Comment:
Fixed it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1544898512
Thanks for the patch @ferdelyi, good job.
Added some comments / questions.
Could you please add testing evidence to the Jira?
To document how this was tested on a real cluster environment.
Also, is there a ZK config page among Hadooop documentation that need to be updated?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on a diff in pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on code in PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#discussion_r1205146014
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java:
##########
@@ -34,31 +39,32 @@
import org.apache.hadoop.fs.CommonConfigurationKeys;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.authentication.util.JaasConfiguration;
+import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.util.ZKUtil;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.client.ZKClientConfig;
+import org.apache.zookeeper.common.ClientX509Util;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Stat;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hadoop.util.Preconditions;
/**
* Helper class that provides utility methods specific to ZK operations.
*/
@InterfaceAudience.Private
public final class ZKCuratorManager {
- private static final Logger LOG =
- LoggerFactory.getLogger(ZKCuratorManager.class);
+ private static final Logger LOG = LoggerFactory.getLogger(ZKCuratorManager.class);
Review Comment:
Thank you for your feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560123778
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 11s | | trunk passed |
| +1 :green_heart: | compile | 16m 50s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 23s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 16m 16s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 16s | | the patch passed |
| +1 :green_heart: | compile | 15m 45s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 15m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 40s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 9s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 28s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 0s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/17/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 183m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 1ec4dbc3c843 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/17/testReport/ |
| Max. process+thread count | 1291 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560122458
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 23s | | trunk passed |
| +1 :green_heart: | compile | 16m 55s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 34s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 16m 11s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 11s | | the patch passed |
| +1 :green_heart: | compile | 15m 41s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 15m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 21s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 58s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/18/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 182m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux dc1f2d4f3a2f 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/18/testReport/ |
| Max. process+thread count | 1906 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560145101
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 37s | | trunk passed |
| +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 26s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 17m 21s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 17m 21s | | the patch passed |
| +1 :green_heart: | compile | 16m 2s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 23s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 0m 54s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/21/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 204m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 09369248f268 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef1756f41ac86ee8522ffab554e16a3237597eea |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/21/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ferdelyi commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "ferdelyi (via GitHub)" <gi...@apache.org>.
ferdelyi commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1560935941
buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5638: HADOOP-18709. Add curator based ZooKeeper communication support over…
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5638:
URL: https://github.com/apache/hadoop/pull/5638#issuecomment-1561271546
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 17s | | trunk passed |
| +1 :green_heart: | compile | 15m 25s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 14m 15s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 14m 56s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 14m 56s | | the patch passed |
| +1 :green_heart: | compile | 14m 21s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 14m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 5s | | hadoop-common in the patch passed. |
| -1 :x: | asflicense | 1m 2s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/22/artifact/out/results-asflicense.txt) | The patch generated 5 ASF License warnings. |
| | | 184m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5638 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs |
| uname | Linux 862e3fca7248 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cf1a0fc110d4d49c7e70c76e98baab0a4686f7dc |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/22/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5638/22/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org