You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/03/03 20:40:41 UTC
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #629:
HDDS-3071. Datanodes unable to connect to recon in Secure Environment
avijayanhwx opened a new pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629
## What changes were proposed in this pull request?
Added Kerberos security between Datanodes and Recon RPC protocol.
Addressed review comments from https://github.com/apache/hadoop-ozone/pull/624 .
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3071
## How was this patch tested?
Manually tested on docker cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387892418
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/ReconDatanodeProtocol.java
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.protocol;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.hdds.recon.ReconConfig;
+import org.apache.hadoop.security.KerberosInfo;
+
+/**
+ * The protocol spoken between datanodes and Recon. For specifics please the
Review comment:
nit: fix typo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387891740
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "This Kerberos principal is used by the Recon service."
+ )
+ private String principal;
+
+ @Config(key = "kerberos.keytab.file",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "The keytab file used by Recon daemon to login as "+
+ "its service principal."
+ )
+ private String keytab;
+
+ @Config(key = "security.client.datanode.container.protocol.acl",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = ""
Review comment:
Can you add a description?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387904236
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
Review comment:
Can we add the newly introduced tag ConfigTag.RECON here for these keys?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387949045
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java
##########
@@ -407,14 +387,48 @@ private static String flatten(String input) {
.replaceAll(" +", " ");
}
- protected String getScmDatanodeAddressKey() {
+ protected RPC.Server getRpcServer(OzoneConfiguration conf,
+ InetSocketAddress datanodeRpcAddr,
Review comment:
@xiaoyuyao I have refactored this a little bit differently. Please let me know if this is OK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387916948
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "This Kerberos principal is used by the Recon service."
+ )
+ private String principal;
+
+ @Config(key = "kerberos.keytab.file",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "The keytab file used by Recon daemon to login as "+
+ "its service principal."
+ )
+ private String keytab;
+
+ @Config(key = "security.client.datanode.container.protocol.acl",
+ type = ConfigType.STRING,
+ defaultValue = "",
Review comment:
Good catch, will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387904716
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "This Kerberos principal is used by the Recon service."
+ )
+ private String principal;
+
+ @Config(key = "kerberos.keytab.file",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "The keytab file used by Recon daemon to login as "+
+ "its service principal."
+ )
+ private String keytab;
+
+ @Config(key = "security.client.datanode.container.protocol.acl",
+ type = ConfigType.STRING,
+ defaultValue = "",
Review comment:
The default value here should be * instead of empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387907777
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
Review comment:
Yes, I can use it. I could not get a general idea of when to use "RECON" config tag. Why do we have "OZONE" config tag? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387910655
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java
##########
@@ -179,21 +179,21 @@ private static void loginReconUser(OzoneConfiguration conf)
if (SecurityUtil.getAuthenticationMethod(conf).equals(
UserGroupInformation.AuthenticationMethod.KERBEROS)) {
-
+ ReconConfig reconConfig = conf.getObject(ReconConfig.class);
LOG.info("Ozone security is enabled. Attempting login for Recon user. "
Review comment:
NIT: "Recon user" => "Recon service"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #629: HDDS-3071.
Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#issuecomment-594157735
cc @vivekratnavel Please review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387905116
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "This Kerberos principal is used by the Recon service."
+ )
+ private String principal;
+
+ @Config(key = "kerberos.keytab.file",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = "The keytab file used by Recon daemon to login as "+
+ "its service principal."
+ )
+ private String keytab;
+
+ @Config(key = "security.client.datanode.container.protocol.acl",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
+ description = ""
+ )
+ private String acl;
Review comment:
Do we miss the getter and setter for the acl?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387908107
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java
##########
@@ -160,22 +161,45 @@ private void addSCMServer(InetSocketAddress address, long sleepTime,
"Ignoring the request.");
return;
}
- RPC.setProtocolEngine(conf, StorageContainerDatanodeProtocolPB.class,
- ProtobufRpcEngine.class);
- long version =
- RPC.getProtocolVersion(StorageContainerDatanodeProtocolPB.class);
-
- RetryPolicy retryPolicy =
- RetryPolicies.retryForeverWithFixedSleep(
- sleepTime, TimeUnit.MILLISECONDS);
- StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
- StorageContainerDatanodeProtocolPB.class, version,
- address, UserGroupInformation.getCurrentUser(), conf,
- NetUtils.getDefaultSocketFactory(conf), getRpcTimeout(),
- retryPolicy).getProxy();
-
- StorageContainerDatanodeProtocolClientSideTranslatorPB rpcClient =
- new StorageContainerDatanodeProtocolClientSideTranslatorPB(rpcProxy);
+
+ StorageContainerDatanodeProtocolClientSideTranslatorPB rpcClient;
+ if (!passiveScm) {
Review comment:
Yes, I agree. Will refactor this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387911372
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconContainerManager.java
##########
@@ -64,6 +69,22 @@ protected File getContainerDBPath(Configuration conf) {
return new File(metaDir, RECON_SCM_CONTAINER_DB);
}
+ public void checkAndAddNewContainer(ContainerID containerID,
+ DatanodeDetails datanodeDetails)
+ throws IOException {
+ if (!exists(containerID)) {
+ LOG.info("New container {} got from {}.", containerID, datanodeDetails);
Review comment:
datanodeDetails toString is very verbose. You may choose to just log the hostname/ip here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387909794
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java
##########
@@ -131,46 +133,24 @@ public SCMDatanodeProtocolServer(final OzoneConfiguration conf,
this.scm = scm;
this.eventPublisher = eventPublisher;
- final int handlerCount =
- conf.getInt(OZONE_SCM_HANDLER_COUNT_KEY,
- OZONE_SCM_HANDLER_COUNT_DEFAULT);
heartbeatDispatcher = new SCMDatanodeHeartbeatDispatcher(
scm.getScmNodeManager(), eventPublisher);
- RPC.setProtocolEngine(conf, StorageContainerDatanodeProtocolPB.class,
Review comment:
NIT: can you highlight the changes relate to the issue? Or the change in this file is a refactor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387894361
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java
##########
@@ -179,21 +179,21 @@ private static void loginReconUser(OzoneConfiguration conf)
if (SecurityUtil.getAuthenticationMethod(conf).equals(
UserGroupInformation.AuthenticationMethod.KERBEROS)) {
-
+ ReconConfig reconConfig = conf.getObject(ReconConfig.class);
LOG.info("Ozone security is enabled. Attempting login for Recon user. "
- + "Principal: {}, keytab: {}", conf.get(
- OZONE_RECON_KERBEROS_PRINCIPAL_KEY),
- conf.get(OZONE_RECON_KERBEROS_KEYTAB_FILE_KEY));
-
+ + "Principal: {}, keytab: {}",
+ reconConfig.getKerberosPrincipal(),
+ reconConfig.getKerberosKeytab());
UserGroupInformation.setConfiguration(conf);
-
InetSocketAddress socAddr = HddsUtils.getReconAddresses(conf);
- SecurityUtil.login(conf, OZONE_RECON_KERBEROS_KEYTAB_FILE_KEY,
- OZONE_RECON_KERBEROS_PRINCIPAL_KEY, socAddr.getHostName());
+ SecurityUtil.login(conf,
+ OZONE_RECON_KERBEROS_KEYTAB_FILE_KEY,
+ OZONE_RECON_KERBEROS_PRINCIPAL_KEY,
+ socAddr.getHostName());
} else {
throw new AuthenticationException(SecurityUtil.getAuthenticationMethod(
- conf) + " authentication method not supported. Recon user login "
- + "failed.");
+ conf) + " authentication method not support. "
Review comment:
nit: supported?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387907041
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java
##########
@@ -160,22 +161,45 @@ private void addSCMServer(InetSocketAddress address, long sleepTime,
"Ignoring the request.");
return;
}
- RPC.setProtocolEngine(conf, StorageContainerDatanodeProtocolPB.class,
- ProtobufRpcEngine.class);
- long version =
- RPC.getProtocolVersion(StorageContainerDatanodeProtocolPB.class);
-
- RetryPolicy retryPolicy =
- RetryPolicies.retryForeverWithFixedSleep(
- sleepTime, TimeUnit.MILLISECONDS);
- StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
- StorageContainerDatanodeProtocolPB.class, version,
- address, UserGroupInformation.getCurrentUser(), conf,
- NetUtils.getDefaultSocketFactory(conf), getRpcTimeout(),
- retryPolicy).getProxy();
-
- StorageContainerDatanodeProtocolClientSideTranslatorPB rpcClient =
- new StorageContainerDatanodeProtocolClientSideTranslatorPB(rpcProxy);
+
+ StorageContainerDatanodeProtocolClientSideTranslatorPB rpcClient;
+ if (!passiveScm) {
Review comment:
Given the reuse part is limited, it is better to move the new logic directly into addReconServer without introducing passiveSCM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #629: HDDS-3071.
Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao merged pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387916168
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java
##########
@@ -407,14 +387,48 @@ private static String flatten(String input) {
.replaceAll(" +", " ");
}
- protected String getScmDatanodeAddressKey() {
+ protected RPC.Server getRpcServer(OzoneConfiguration conf,
+ InetSocketAddress datanodeRpcAddr,
Review comment:
NIT: If we add a parameter Class<?> protocol for protocol? we don't need to overwrite getRpcServer in ReconDatanodeProtocolServer class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #629: HDDS-3071. Datanodes unable to connect to recon in Secure Environment
URL: https://github.com/apache/hadoop-ozone/pull/629#discussion_r387907777
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon/ReconConfig.java
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.recon;
+
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.ConfigTag;
+import org.apache.hadoop.hdds.conf.ConfigType;
+
+/**
+ * The configuration class for the Recon service.
+ */
+@ConfigGroup(prefix = "ozone.recon")
+public class ReconConfig {
+
+ @Config(key = "kerberos.principal",
+ type = ConfigType.STRING,
+ defaultValue = "",
+ tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
Review comment:
Yes, I can use it. I could not get a general idea of when to use "RECON" config tag. Why do we have an "OZONE" config tag? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org