You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/09/04 21:21:48 UTC
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
bharatviswa504 opened a new pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398
## What changes were proposed in this pull request?
When bucket links are levels, When ACLS are enabled, during `checkAcls` It will fail with NPE.
Finally in HA, this causes OM to be terminated.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4210
## How was this patch tested?
Added UT.
The same test will fail without the fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r488047498
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh
##########
@@ -30,6 +30,8 @@ execute_robot_test scm kinit.robot
execute_robot_test scm freon
+execute_robot_test scm basic/links.robot
Review comment:
Makes sense. Thanks for all your clarification!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487831535
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3479,10 +3481,35 @@ public ResolvedBucket resolveBucketLink(OmKeyArgs args)
Pair.of(args.getVolumeName(), args.getBucketName()));
}
+ public ResolvedBucket resolveBucketLink(Pair<String, String> requested,
+ OMClientRequest omClientRequest)
+ throws IOException {
+ Pair<String, String> resolved;
+ if (isAclEnabled) {
+ resolved = resolveBucketLink(requested, new HashSet<>(),
+ omClientRequest.createUGI(), omClientRequest.getRemoteAddress(),
+ omClientRequest.getHostName());
+ } else {
+ resolved = resolveBucketLink(requested, new HashSet<>(),
+ null, null, null);
+ }
+ return new ResolvedBucket(requested, resolved);
+ }
+
+
public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
throws IOException {
- Pair<String, String> resolved =
- resolveBucketLink(requested, new HashSet<>());
+
+ Pair<String, String> resolved;
+ if (isAclEnabled) {
+ resolved = resolveBucketLink(requested, new HashSet<>(),
+ ProtobufRpcEngine.Server.getRemoteUser(),
+ ProtobufRpcEngine.Server.getRemoteIp(),
+ ProtobufRpcEngine.Server.getRemoteIp().getHostName());
Review comment:
Nit: to avoid Sonar warning that _"static" base class members should not be accessed via derived types_.
```suggestion
Server.getRemoteUser(),
Server.getRemoteIp(),
Server.getRemoteIp().getHostName());
```
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r483929995
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
We already have acceptance test for links, but currently it is only run in non-HA setup. Instead of adding integration test, I think we should run the existing test in HA setup, too:
```
execute_robot_test scm basic/links.robot
```
could be added to `hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh` and maybe `hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/test.sh`.
This confirms the bug and the fix.
However, I also see two of the test cases failing with the fix:
```
Cannot follow link without read access | FAIL |
ACL verified on source bucket | FAIL |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r485100541
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
+1 for that. Also suggest separate related members from OMCleintRequest (HA) before passing them to OzoneManager (non-HA).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r483929995
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
We already have acceptance test for links, but currently it is only run in non-HA setup. Instead of adding integration test, I think we should run the existing test in HA setup, too:
```
execute_robot_test scm basic/links.robot
```
could be added to `hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh` and maybe `hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/test.sh`.
This confirms that the bug and the fix.
However, I also see two of the test cases failing with the fix:
```
Cannot follow link without read access | FAIL |
ACL verified on source bucket | FAIL |
```
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Would it be possible to replace the original `resolveBucketLink` method with this one, instead of duplicating it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r486559825
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
I have taken this approach of passing OMClientRequest so that we don't need to touch all OMKey write Requests in passing this info. (With this only class needs to be updated which is base class OMKeyRequest)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487634517
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh
##########
@@ -30,6 +30,8 @@ execute_robot_test scm kinit.robot
execute_robot_test scm freon
+execute_robot_test scm basic/links.robot
Review comment:
Good question might be it is good to run only during security-enabled cluster.
But it has some tests which are bucket link features, so I thought it might be good to run in both secure and non-secure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487178258
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487178258
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487468731
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh
##########
@@ -30,6 +30,8 @@ execute_robot_test scm kinit.robot
execute_robot_test scm freon
+execute_robot_test scm basic/links.robot
Review comment:
Out of curiosity: why adding this robot test to both
` hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/test.sh` and
`hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh`.
Will it just run the same test twice on each CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r486559825
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
I have taken this approach of passing OMClientRequest so that we don't need to touch all OMKey write Requests in passing this info. (With this only class needs to be updated which is base class OMKeyRequest)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r486559825
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
I have taken this approach of passing OMClientRequest so that we don't need to touch all OMKey write Requests in passing this info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#issuecomment-692396188
@xiaoyuyao Do you have any more comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487181977
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithData.java
##########
@@ -69,6 +70,38 @@ public void testAllOMNodesRunning() throws Exception {
createKeyTest(true);
}
+ @Test
+ public void testBucketLinkOps() throws Exception {
Review comment:
Thanks for the tip. Enabled docker-compose tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#issuecomment-693141553
Review comments are addressed, and no further comments, proceeding with commit.
Thank You @adoroszlai and @xiaoyuyao for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487178258
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Made few changes to not to duplicate code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487834748
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3523,6 +3533,47 @@ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
visited);
}
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param {@link OMClientRequest} which has information required to check
+ * permission.
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited,
+ OMClientRequest omClientRequest) throws IOException {
Review comment:
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1398:
URL: https://github.com/apache/hadoop-ozone/pull/1398#discussion_r487700177
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/test.sh
##########
@@ -30,6 +30,8 @@ execute_robot_test scm kinit.robot
execute_robot_test scm freon
+execute_robot_test scm basic/links.robot
Review comment:
I think it's useful to run the same test in different environments: secure and non-secure, non-HA and HA. It increases coverage. Also, non-secure versions provide some kind of baseline: if it works and secure one does not, we might know the problem is security-related.
If it takes too much time, we can tweak acceptance test splits (currently we have 3 concurrent runs in CI).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org