You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/07/01 14:17:41 UTC
[GitHub] [ozone] smengcl opened a new pull request, #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
smengcl opened a new pull request, #3576:
URL: https://github.com/apache/ozone/pull/3576
## What changes were proposed in this pull request?
Use `RangerClient` for Ranger read and write operations.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6909
## How was this patch tested?
- [ ] All existing tests should pass.
- [x] `TestRangerBGSyncService` passed when connected to a remote Ranger Admin Server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919404136
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -267,6 +267,11 @@ public static boolean isReadOnly(
case ListTenant:
case TenantGetUserInfo:
case TenantListUser:
+ case RangerBGSync:
+ // RangerBGSync is a read operation in the sense that it doesn't directly
+ // write to OM DB. And therefore it doesn't need a OMClientRequest.
+ // Although indirectly the Ranger sync service task could invoke write
Review Comment:
Right. It could indirectly trigger another write operation. But for the implementation `RangerBGSync` should be listed under the `readonly` list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919400786
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -267,6 +267,11 @@ public static boolean isReadOnly(
case ListTenant:
case TenantGetUserInfo:
case TenantListUser:
+ case RangerBGSync:
+ // RangerBGSync is a read operation in the sense that it doesn't directly
+ // write to OM DB. And therefore it doesn't need a OMClientRequest.
+ // Although indirectly the Ranger sync service task could invoke write
Review Comment:
Nit: This implies it should not be read-only..?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919560920
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -3081,6 +3083,53 @@ public ServiceInfoEx getServiceInfo() throws IOException {
return new ServiceInfoEx(getServiceList(), caCertPem, caCertPemList);
}
+ @Override
+ public boolean triggerRangerBGSync(boolean noWait) throws IOException {
+
+ // OM should be leader and ready.
+ // This check should always pass if called from a client since there
+ // is a leader check somewhere before entering this method.
+ if (!isLeaderReady()) {
+ // And even if we could allow followers to trigger sync, checkLeader()
+ // calls inside the sync would quit the sync anyway.
+ throw new OMException("OM is not leader or not ready", INVALID_REQUEST);
+ }
+
+ final UserGroupInformation ugi = getRemoteUser();
+ // Check Ozone admin privilege
+ if (!isAdmin(ugi)) {
+ throw new OMException("Only Ozone admins are allowed to trigger "
+ + "Ranger background sync manually", PERMISSION_DENIED);
+ }
+
+ // Check if MT manager is inited
+ final OMMultiTenantManager mtManager = getMultiTenantManager();
+ if (mtManager == null) {
+ throw new OMException("S3 Multi-Tenancy is not enabled",
+ FEATURE_NOT_ENABLED);
+ }
+
+ // Check if Ranger BG sync task is inited
+ final OMRangerBGSyncService bgSync = mtManager.getOMRangerBGSyncService();
+ if (bgSync == null) {
+ throw new OMException("Background sync task is not initialized",
+ FEATURE_NOT_ENABLED);
+ }
+
+ // Trigger Ranger BG Sync
+ if (noWait) {
+ final Thread t = new Thread(bgSync::triggerRangerSyncOnce);
+ t.start();
+ LOG.info("User '{}' manually triggered Multi-Tenancy Ranger Sync "
+ + "in a new thread, tid={}", ugi, t.getId());
+ return true;
+ } else {
Review Comment:
Yup. Though I personally like to keep this for symmetry. There is not too much code in the `else` part. If it grows larger I would unindent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919459614
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
Review Comment:
Is there a longer term plan for using customer provided trust chain?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on PR #3576:
URL: https://github.com/apache/ozone/pull/3576#issuecomment-1182662845
> Most of my comments are minor nits or future work, the CI is failing in findbugs
>
> ```
> M D SF: Switch statement found in org.apache.hadoop.ozone.om.multitenant.RangerClientMultiTenantAccessController.decodeRSEStatusCodes(RangerServiceException) where default case is missing At RangerClientMultiTenantAccessController.java:[lines 152-157]
> ```
>
> Once it passes we can merge this change in.
Thanks @kerneltime for the review. I have addressed the comments so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919561906
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
Review Comment:
Oh. This part is only in the test code and not used in production, and was copied from our earlier homemade Ranger client. This could be useful during dev, though the proper way would be to set a correct truststore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3576:
URL: https://github.com/apache/ozone/pull/3576#issuecomment-1182585023
Most of my comments are minor nits or future work, the CI is failing in findbugs
```
M D SF: Switch statement found in org.apache.hadoop.ozone.om.multitenant.RangerClientMultiTenantAccessController.decodeRSEStatusCodes(RangerServiceException) where default case is missing At RangerClientMultiTenantAccessController.java:[lines 152-157]
```
Once it passes we can merge this change in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919421693
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerClientMultiTenantAccessController.java:
##########
@@ -56,138 +60,276 @@ public class RangerClientMultiTenantAccessController implements
private static final Logger LOG = LoggerFactory
.getLogger(RangerClientMultiTenantAccessController.class);
+ private static final int HTTP_STATUS_CODE_UNAUTHORIZED = 401;
+ private static final int HTTP_STATUS_CODE_BAD_REQUEST = 400;
+
private final RangerClient client;
private final String rangerServiceName;
private final Map<IAccessAuthorizer.ACLType, String> aclToString;
private final Map<String, IAccessAuthorizer.ACLType> stringToAcl;
private final String omPrincipal;
+ // execUser for Ranger
+ private final String shortName;
public RangerClientMultiTenantAccessController(OzoneConfiguration conf)
throws IOException {
+
aclToString = MultiTenantAccessController.getRangerAclStrings();
stringToAcl = new HashMap<>();
aclToString.forEach((type, string) -> stringToAcl.put(string, type));
- // Should have passed the check in OMMultiTenantManager
+ // Should have passed the config checks in
+ // OMMultiTenantManager#checkAndEnableMultiTenancy at this point.
+
String rangerHttpsAddress = conf.get(OZONE_RANGER_HTTPS_ADDRESS_KEY);
Preconditions.checkNotNull(rangerHttpsAddress);
rangerServiceName = conf.get(OZONE_RANGER_SERVICE);
Preconditions.checkNotNull(rangerServiceName);
- String configuredOmPrincipal = conf.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY);
- Preconditions.checkNotNull(configuredOmPrincipal);
- // Replace _HOST pattern with host name in the Kerberos principal. Ranger
- // client currently does not do this automatically.
- omPrincipal = SecurityUtil.getServerPrincipal(
- configuredOmPrincipal, OmUtils.getOmAddress(conf).getHostName());
- String keytabPath = conf.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY);
- Preconditions.checkNotNull(keytabPath);
+ // Determine auth type (KERBEROS or SIMPLE)
+ final String authType;
+ final String usernameOrPrincipal;
+ final String passwordOrKeytab;
+
+ // If both OZONE_OM_RANGER_HTTPS_ADMIN_API_USER and
+ // OZONE_OM_RANGER_HTTPS_ADMIN_API_PASSWD are set, SIMPLE auth will be used
+ String fallbackUsername = conf.get(OZONE_OM_RANGER_HTTPS_ADMIN_API_USER);
+ String fallbackPassword = conf.get(OZONE_OM_RANGER_HTTPS_ADMIN_API_PASSWD);
+
+ if (fallbackUsername != null && fallbackPassword != null) {
+ // Both clear text username and password are set, use SIMPLE auth.
+ authType = AuthenticationMethod.SIMPLE.name();
+
+ usernameOrPrincipal = fallbackUsername;
+ passwordOrKeytab = fallbackPassword;
+
+ omPrincipal = fallbackUsername;
+ shortName = fallbackUsername;
+ } else {
+ // Use KERBEROS auth.
+ authType = AuthenticationMethod.KERBEROS.name();
+
+ String configuredOmPrincipal = conf.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY);
+ Preconditions.checkNotNull(configuredOmPrincipal);
+
+ // Replace _HOST pattern with host name in the Kerberos principal.
+ // Ranger client currently does not do this automatically.
+ omPrincipal = SecurityUtil.getServerPrincipal(
+ configuredOmPrincipal, OmUtils.getOmAddress(conf).getHostName());
+ final String keytabPath = conf.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY);
+ Preconditions.checkNotNull(keytabPath);
+
+ // Convert to short name to be used in some Ranger requests
+ shortName = UserGroupInformation.createRemoteUser(omPrincipal)
+ .getShortUserName();
+
+ usernameOrPrincipal = omPrincipal;
+ passwordOrKeytab = keytabPath;
+ }
+
+ LOG.info("authType = {}, login user = {}", authType, usernameOrPrincipal);
client = new RangerClient(rangerHttpsAddress,
- KERBEROS.name().toLowerCase(), omPrincipal, keytabPath,
+ authType, usernameOrPrincipal, passwordOrKeytab,
rangerServiceName, OzoneConsts.OZONE);
+
+ // Whether or not the Ranger credentials are valid is unknown right after
+ // RangerClient initialization here. Because RangerClient does not perform
+ // any authentication at this point just yet.
+ //
+ // If the credentials are invalid, RangerClient later throws 401 in every
+ // single request to Ranger.
+ }
+
+ /**
+ * Check StatusCode from RangerServiceException and try to log helpful,
+ * actionable messages.
+ *
+ * @param rse RangerServiceException
+ */
+ private void decodeRSEStatusCodes(RangerServiceException rse) {
+
+ switch (rse.getStatus().getStatusCode()) {
+ case HTTP_STATUS_CODE_UNAUTHORIZED:
+ LOG.error("Auth failure. Please double check Ranger-related configs");
+ break;
+ case HTTP_STATUS_CODE_BAD_REQUEST:
+ LOG.error("Request failure. If this is an assign-user operation, "
+ + "check if the user name exists in Ranger.");
+ default:
Review Comment:
Nit: Do we want to log return codes other than 200?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919453501
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -3081,6 +3083,53 @@ public ServiceInfoEx getServiceInfo() throws IOException {
return new ServiceInfoEx(getServiceList(), caCertPem, caCertPemList);
}
+ @Override
+ public boolean triggerRangerBGSync(boolean noWait) throws IOException {
+
+ // OM should be leader and ready.
+ // This check should always pass if called from a client since there
+ // is a leader check somewhere before entering this method.
+ if (!isLeaderReady()) {
+ // And even if we could allow followers to trigger sync, checkLeader()
+ // calls inside the sync would quit the sync anyway.
+ throw new OMException("OM is not leader or not ready", INVALID_REQUEST);
+ }
+
+ final UserGroupInformation ugi = getRemoteUser();
+ // Check Ozone admin privilege
+ if (!isAdmin(ugi)) {
+ throw new OMException("Only Ozone admins are allowed to trigger "
+ + "Ranger background sync manually", PERMISSION_DENIED);
+ }
+
+ // Check if MT manager is inited
+ final OMMultiTenantManager mtManager = getMultiTenantManager();
+ if (mtManager == null) {
+ throw new OMException("S3 Multi-Tenancy is not enabled",
+ FEATURE_NOT_ENABLED);
+ }
+
+ // Check if Ranger BG sync task is inited
+ final OMRangerBGSyncService bgSync = mtManager.getOMRangerBGSyncService();
+ if (bgSync == null) {
+ throw new OMException("Background sync task is not initialized",
+ FEATURE_NOT_ENABLED);
+ }
+
+ // Trigger Ranger BG Sync
+ if (noWait) {
+ final Thread t = new Thread(bgSync::triggerRangerSyncOnce);
+ t.start();
+ LOG.info("User '{}' manually triggered Multi-Tenancy Ranger Sync "
+ + "in a new thread, tid={}", ugi, t.getId());
+ return true;
+ } else {
Review Comment:
Minor Nit: No else needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919564931
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
+ TrustManager[] trustAllCerts = new TrustManager[]{
+ new X509TrustManager() {
+ public java.security.cert.X509Certificate[] getAcceptedIssuers() {
+ return null;
+ }
+ public void checkClientTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ public void checkServerTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ }
+ };
+
+ try {
+ SSLContext sc = SSLContext.getInstance("SSL");
+ sc.init(null, trustAllCerts, new java.security.SecureRandom());
+ HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
+ } catch (Exception e) {
+ LOG.info("Setting DefaultSSLSocketFactory failed.");
+ }
+ }
+
+ private HttpURLConnection openURLConnection(URL url) throws IOException {
+ final HttpURLConnection urlConnection;
+ if (url.getProtocol().equals("https")) {
+ urlConnection = (HttpsURLConnection) url.openConnection();
+ } else if (url.getProtocol().equals("http")) {
+ urlConnection = (HttpURLConnection) url.openConnection();
+ } else {
+ throw new IOException("Unsupported protocol: " + url.getProtocol() +
+ "URL: " + url);
+ }
+ return urlConnection;
+ }
+
+ /**
+ * Can make either http or https request.
+ */
+ private HttpURLConnection makeHttpCall(String urlString,
+ String jsonInputString, String method, boolean isSpnego)
+ throws IOException {
+
+ URL url = new URL(urlString);
+ final HttpURLConnection urlConnection = openURLConnection(url);
+
+ urlConnection.setRequestMethod(method);
+ // Timeout in ms
+ urlConnection.setConnectTimeout(5000);
Review Comment:
Added in 72a4d60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime merged pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime merged PR #3576:
URL: https://github.com/apache/ozone/pull/3576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919562524
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
+ TrustManager[] trustAllCerts = new TrustManager[]{
+ new X509TrustManager() {
+ public java.security.cert.X509Certificate[] getAcceptedIssuers() {
+ return null;
+ }
+ public void checkClientTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ public void checkServerTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ }
+ };
+
+ try {
+ SSLContext sc = SSLContext.getInstance("SSL");
+ sc.init(null, trustAllCerts, new java.security.SecureRandom());
+ HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
+ } catch (Exception e) {
+ LOG.info("Setting DefaultSSLSocketFactory failed.");
+ }
+ }
+
+ private HttpURLConnection openURLConnection(URL url) throws IOException {
+ final HttpURLConnection urlConnection;
+ if (url.getProtocol().equals("https")) {
+ urlConnection = (HttpsURLConnection) url.openConnection();
+ } else if (url.getProtocol().equals("http")) {
+ urlConnection = (HttpURLConnection) url.openConnection();
+ } else {
+ throw new IOException("Unsupported protocol: " + url.getProtocol() +
+ "URL: " + url);
+ }
+ return urlConnection;
+ }
+
+ /**
+ * Can make either http or https request.
+ */
+ private HttpURLConnection makeHttpCall(String urlString,
+ String jsonInputString, String method, boolean isSpnego)
+ throws IOException {
+
+ URL url = new URL(urlString);
+ final HttpURLConnection urlConnection = openURLConnection(url);
+
+ urlConnection.setRequestMethod(method);
+ // Timeout in ms
+ urlConnection.setConnectTimeout(5000);
Review Comment:
Yes. The timeout in production code is tunable (though `RangerRestMultiTenantAccessController` is unused and I might even remove it at a later point. But keeping it now so we have a choice if `RangerClient` is not working): https://github.com/apache/ozone/blob/fc3a23af25d61146139ad434b2323bbeedc89828/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerRestMultiTenantAccessController.java#L440
`RangerUserRequest` is test-only code so I didn't bother. I could read the some config as well though.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
+ TrustManager[] trustAllCerts = new TrustManager[]{
+ new X509TrustManager() {
+ public java.security.cert.X509Certificate[] getAcceptedIssuers() {
+ return null;
+ }
+ public void checkClientTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ public void checkServerTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ }
+ };
+
+ try {
+ SSLContext sc = SSLContext.getInstance("SSL");
+ sc.init(null, trustAllCerts, new java.security.SecureRandom());
+ HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
+ } catch (Exception e) {
+ LOG.info("Setting DefaultSSLSocketFactory failed.");
+ }
+ }
+
+ private HttpURLConnection openURLConnection(URL url) throws IOException {
+ final HttpURLConnection urlConnection;
+ if (url.getProtocol().equals("https")) {
+ urlConnection = (HttpsURLConnection) url.openConnection();
+ } else if (url.getProtocol().equals("http")) {
+ urlConnection = (HttpURLConnection) url.openConnection();
+ } else {
+ throw new IOException("Unsupported protocol: " + url.getProtocol() +
+ "URL: " + url);
+ }
+ return urlConnection;
+ }
+
+ /**
+ * Can make either http or https request.
+ */
+ private HttpURLConnection makeHttpCall(String urlString,
+ String jsonInputString, String method, boolean isSpnego)
+ throws IOException {
+
+ URL url = new URL(urlString);
+ final HttpURLConnection urlConnection = openURLConnection(url);
+
+ urlConnection.setRequestMethod(method);
+ // Timeout in ms
+ urlConnection.setConnectTimeout(5000);
Review Comment:
Yes. The timeout in production code is tunable (though `RangerRestMultiTenantAccessController` is unused and I might even remove it at a later point. But keeping it now so we have a choice if `RangerClient` is not working): https://github.com/apache/ozone/blob/fc3a23af25d61146139ad434b2323bbeedc89828/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerRestMultiTenantAccessController.java#L440
`RangerUserRequest` is test-only code so I didn't bother. I could read the same config as well though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919562524
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
+ TrustManager[] trustAllCerts = new TrustManager[]{
+ new X509TrustManager() {
+ public java.security.cert.X509Certificate[] getAcceptedIssuers() {
+ return null;
+ }
+ public void checkClientTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ public void checkServerTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ }
+ };
+
+ try {
+ SSLContext sc = SSLContext.getInstance("SSL");
+ sc.init(null, trustAllCerts, new java.security.SecureRandom());
+ HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
+ } catch (Exception e) {
+ LOG.info("Setting DefaultSSLSocketFactory failed.");
+ }
+ }
+
+ private HttpURLConnection openURLConnection(URL url) throws IOException {
+ final HttpURLConnection urlConnection;
+ if (url.getProtocol().equals("https")) {
+ urlConnection = (HttpsURLConnection) url.openConnection();
+ } else if (url.getProtocol().equals("http")) {
+ urlConnection = (HttpURLConnection) url.openConnection();
+ } else {
+ throw new IOException("Unsupported protocol: " + url.getProtocol() +
+ "URL: " + url);
+ }
+ return urlConnection;
+ }
+
+ /**
+ * Can make either http or https request.
+ */
+ private HttpURLConnection makeHttpCall(String urlString,
+ String jsonInputString, String method, boolean isSpnego)
+ throws IOException {
+
+ URL url = new URL(urlString);
+ final HttpURLConnection urlConnection = openURLConnection(url);
+
+ urlConnection.setRequestMethod(method);
+ // Timeout in ms
+ urlConnection.setConnectTimeout(5000);
Review Comment:
Yes. The actual production timeout is tunable: https://github.com/apache/ozone/blob/fc3a23af25d61146139ad434b2323bbeedc89828/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerRestMultiTenantAccessController.java#L440
`RangerUserRequest` is test-only code so I didn't bother. I could read the some config as well though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919558295
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerClientMultiTenantAccessController.java:
##########
@@ -56,138 +60,276 @@ public class RangerClientMultiTenantAccessController implements
private static final Logger LOG = LoggerFactory
.getLogger(RangerClientMultiTenantAccessController.class);
+ private static final int HTTP_STATUS_CODE_UNAUTHORIZED = 401;
+ private static final int HTTP_STATUS_CODE_BAD_REQUEST = 400;
+
private final RangerClient client;
private final String rangerServiceName;
private final Map<IAccessAuthorizer.ACLType, String> aclToString;
private final Map<String, IAccessAuthorizer.ACLType> stringToAcl;
private final String omPrincipal;
+ // execUser for Ranger
+ private final String shortName;
public RangerClientMultiTenantAccessController(OzoneConfiguration conf)
throws IOException {
+
aclToString = MultiTenantAccessController.getRangerAclStrings();
stringToAcl = new HashMap<>();
aclToString.forEach((type, string) -> stringToAcl.put(string, type));
- // Should have passed the check in OMMultiTenantManager
+ // Should have passed the config checks in
+ // OMMultiTenantManager#checkAndEnableMultiTenancy at this point.
+
String rangerHttpsAddress = conf.get(OZONE_RANGER_HTTPS_ADDRESS_KEY);
Preconditions.checkNotNull(rangerHttpsAddress);
rangerServiceName = conf.get(OZONE_RANGER_SERVICE);
Preconditions.checkNotNull(rangerServiceName);
- String configuredOmPrincipal = conf.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY);
- Preconditions.checkNotNull(configuredOmPrincipal);
- // Replace _HOST pattern with host name in the Kerberos principal. Ranger
- // client currently does not do this automatically.
- omPrincipal = SecurityUtil.getServerPrincipal(
- configuredOmPrincipal, OmUtils.getOmAddress(conf).getHostName());
- String keytabPath = conf.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY);
- Preconditions.checkNotNull(keytabPath);
+ // Determine auth type (KERBEROS or SIMPLE)
+ final String authType;
+ final String usernameOrPrincipal;
+ final String passwordOrKeytab;
+
+ // If both OZONE_OM_RANGER_HTTPS_ADMIN_API_USER and
+ // OZONE_OM_RANGER_HTTPS_ADMIN_API_PASSWD are set, SIMPLE auth will be used
+ String fallbackUsername = conf.get(OZONE_OM_RANGER_HTTPS_ADMIN_API_USER);
+ String fallbackPassword = conf.get(OZONE_OM_RANGER_HTTPS_ADMIN_API_PASSWD);
+
+ if (fallbackUsername != null && fallbackPassword != null) {
+ // Both clear text username and password are set, use SIMPLE auth.
+ authType = AuthenticationMethod.SIMPLE.name();
+
+ usernameOrPrincipal = fallbackUsername;
+ passwordOrKeytab = fallbackPassword;
+
+ omPrincipal = fallbackUsername;
+ shortName = fallbackUsername;
+ } else {
+ // Use KERBEROS auth.
+ authType = AuthenticationMethod.KERBEROS.name();
+
+ String configuredOmPrincipal = conf.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY);
+ Preconditions.checkNotNull(configuredOmPrincipal);
+
+ // Replace _HOST pattern with host name in the Kerberos principal.
+ // Ranger client currently does not do this automatically.
+ omPrincipal = SecurityUtil.getServerPrincipal(
+ configuredOmPrincipal, OmUtils.getOmAddress(conf).getHostName());
+ final String keytabPath = conf.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY);
+ Preconditions.checkNotNull(keytabPath);
+
+ // Convert to short name to be used in some Ranger requests
+ shortName = UserGroupInformation.createRemoteUser(omPrincipal)
+ .getShortUserName();
+
+ usernameOrPrincipal = omPrincipal;
+ passwordOrKeytab = keytabPath;
+ }
+
+ LOG.info("authType = {}, login user = {}", authType, usernameOrPrincipal);
client = new RangerClient(rangerHttpsAddress,
- KERBEROS.name().toLowerCase(), omPrincipal, keytabPath,
+ authType, usernameOrPrincipal, passwordOrKeytab,
rangerServiceName, OzoneConsts.OZONE);
+
+ // Whether or not the Ranger credentials are valid is unknown right after
+ // RangerClient initialization here. Because RangerClient does not perform
+ // any authentication at this point just yet.
+ //
+ // If the credentials are invalid, RangerClient later throws 401 in every
+ // single request to Ranger.
+ }
+
+ /**
+ * Check StatusCode from RangerServiceException and try to log helpful,
+ * actionable messages.
+ *
+ * @param rse RangerServiceException
+ */
+ private void decodeRSEStatusCodes(RangerServiceException rse) {
+
+ switch (rse.getStatus().getStatusCode()) {
+ case HTTP_STATUS_CODE_UNAUTHORIZED:
+ LOG.error("Auth failure. Please double check Ranger-related configs");
+ break;
+ case HTTP_STATUS_CODE_BAD_REQUEST:
+ LOG.error("Request failure. If this is an assign-user operation, "
+ + "check if the user name exists in Ranger.");
+ default:
Review Comment:
This is a helper function to translate the return code during error because I find the non-200 status codes often unintuitive from Ranger. (And Ranger sometimes even masks the real reason behind a failure in some operations / cases.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3576: HDDS-6909. [Multi-Tenant] Use RangerClient for Ranger operations
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3576:
URL: https://github.com/apache/ozone/pull/3576#discussion_r919459909
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/RangerUserRequest.java:
##########
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.multitenant;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
+import org.apache.kerby.util.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+import static java.net.HttpURLConnection.HTTP_OK;
+// TODO: MOVE THIS HERE.
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_CREATE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_DELETE_USER_HTTP_ENDPOINT;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_OM_RANGER_ADMIN_GET_USER_HTTP_ENDPOINT;
+
+/**
+ * Helper class to create and delete users in Ranger because RangerClient
+ * doesn't support it yet.
+ */
+public class RangerUserRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangerUserRequest.class);
+
+ private String rangerEndpoint;
+
+ // Value of HTTP auth header
+ private String authHeaderValue;
+
+ RangerUserRequest(String rangerHttpsAddress, String userName, String passwd) {
+
+ // Trim trailing slash
+ if (rangerHttpsAddress.endsWith("/")) {
+ rangerHttpsAddress =
+ rangerHttpsAddress.substring(0, rangerHttpsAddress.length() - 1);
+ }
+ this.rangerEndpoint = rangerHttpsAddress;
+
+ String auth = userName + ":" + passwd;
+ byte[] encodedAuth =
+ Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
+ authHeaderValue = "Basic " +
+ new String(encodedAuth, StandardCharsets.UTF_8);
+
+ setupRangerIgnoreServerCertificate();
+ }
+
+ private void setupRangerIgnoreServerCertificate() {
+ // Create a trust manager that does not validate certificate chains
+ TrustManager[] trustAllCerts = new TrustManager[]{
+ new X509TrustManager() {
+ public java.security.cert.X509Certificate[] getAcceptedIssuers() {
+ return null;
+ }
+ public void checkClientTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ public void checkServerTrusted(
+ java.security.cert.X509Certificate[] certs, String authType) {
+ }
+ }
+ };
+
+ try {
+ SSLContext sc = SSLContext.getInstance("SSL");
+ sc.init(null, trustAllCerts, new java.security.SecureRandom());
+ HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
+ } catch (Exception e) {
+ LOG.info("Setting DefaultSSLSocketFactory failed.");
+ }
+ }
+
+ private HttpURLConnection openURLConnection(URL url) throws IOException {
+ final HttpURLConnection urlConnection;
+ if (url.getProtocol().equals("https")) {
+ urlConnection = (HttpsURLConnection) url.openConnection();
+ } else if (url.getProtocol().equals("http")) {
+ urlConnection = (HttpURLConnection) url.openConnection();
+ } else {
+ throw new IOException("Unsupported protocol: " + url.getProtocol() +
+ "URL: " + url);
+ }
+ return urlConnection;
+ }
+
+ /**
+ * Can make either http or https request.
+ */
+ private HttpURLConnection makeHttpCall(String urlString,
+ String jsonInputString, String method, boolean isSpnego)
+ throws IOException {
+
+ URL url = new URL(urlString);
+ final HttpURLConnection urlConnection = openURLConnection(url);
+
+ urlConnection.setRequestMethod(method);
+ // Timeout in ms
+ urlConnection.setConnectTimeout(5000);
Review Comment:
ToDo: Make timeout values configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org