You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/11/09 23:58:51 UTC
[GitHub] [ozone] aryangupta1998 opened a new pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
aryangupta1998 opened a new pull request #2819:
URL: https://github.com/apache/ozone/pull/2819
## What changes were proposed in this pull request?
Presently FILE_SYSTEM_OPTIMIZED("FSO") and OBJECT_STORE("OBS") implementation depends on "ozone.om.enable.filesystem.paths" flag. This flag should be enabled explicitly for the FSO logic and disabled explicitly for the OBS logic. This task is to remove the dependency with the normalization config flag for the FSO and OBS bucket layouts.
case-1) FSO bucket layout should normalize the given key path irrespective of the above flag value.
case-2) OBS bucket layout shouldn't normalize the given key path irrespective of the above flag value.
case-3) LEGACY bucket layout should read the configuration flag and perform normalize the given key path based on the above config flag.
Hint
`// normalize key if required.`
`String keyName;`
` if (getBucketLayout() == BucketLayout.OBJECT_STORE) {`
`keyName = keyArgs.getKeyName(); // pure object store semantics.`
`} else if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED) {`
`keyName = validateAndNormalizeKey(true, keyArgs.getKeyName()); // pure FS semantics.`
` } else {`
` // legacy bucket, normalize key based on "ozone.om.enable.filesystem.paths" flag value.`
`keyName = validateAndNormalizeKey(`
` ozoneManager.getEnableFileSystemPaths(), keyArgs.getKeyName());`
` }`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5929
## How was this patch tested?
Tested Manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#issuecomment-964818580
@aryangupta1998
Test case failures are due to a special case `OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS, true`.
TestOzoneFSWithObjectStoreCreate and TestOzoneFileSystemMetrics.
These tests expects the keyPath to be normalized but at the OM side the default bucket layout has been set to OBS layout, which won't do normalization now and resulted in failure.
Please do two changes to fix the test failure:
(1) Set `conf.set(OMConfigKeys.OZONE_DEFAULT_BUCKET_LAYOUT, BucketLayout.LEGACY.name())`; to the test case. Also, the same has to be set in the robot test cases(acceptance tests as well).
(2) Add LEGACY to the allowed values for the ozone.default.bucket.layout config at OzoneManager.java class. @JyotinderSingh also doing this change in HDDS-5839, so we can take an incremental approach now and gradually will remove the LEGACY support in future.
```
// Make sure defaultBucketLayout is set to a valid value
if (!defaultBucketLayout.equals(
BucketLayout.FILE_SYSTEM_OPTIMIZED.name()) &&
!defaultBucketLayout.equals(BucketLayout.OBJECT_STORE.name())
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a change in pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#discussion_r746781404
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
##########
@@ -78,6 +80,29 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
Preconditions.checkNotNull(allocateBlockRequest);
KeyArgs keyArgs = allocateBlockRequest.getKeyArgs();
+ String keyPath = keyArgs.getKeyName();
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#issuecomment-964872926
@aryangupta1998
> (2) Add LEGACY to the allowed values for the ozone.default.bucket.layout config at OzoneManager.java class. @JyotinderSingh also doing this change in [HDDS-5839](https://issues.apache.org/jira/browse/HDDS-5839), so we can take an incremental approach now and gradually will remove the LEGACY support in future.
[HDDS-5839](https://issues.apache.org/jira/browse/HDDS-5839) has been merged, so please include that changes by rebasing your PR branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a change in pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#discussion_r746678202
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
##########
@@ -78,6 +80,29 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
Preconditions.checkNotNull(allocateBlockRequest);
KeyArgs keyArgs = allocateBlockRequest.getKeyArgs();
+ String keyPath = keyArgs.getKeyName();
+ BucketLayout bucketLayout = getBucketLayout();
+ LOG.info("Bucket Layout: " + bucketLayout);
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#issuecomment-965991199
+1 LGTM. Thanks @aryangupta1998 for addressing all the comments. I will merge it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a change in pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#discussion_r746259819
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
##########
@@ -78,6 +80,29 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
Preconditions.checkNotNull(allocateBlockRequest);
KeyArgs keyArgs = allocateBlockRequest.getKeyArgs();
+ String keyPath = keyArgs.getKeyName();
+ BucketLayout bucketLayout = getBucketLayout();
+ LOG.info("Bucket Layout: " + bucketLayout);
Review comment:
Please change the priority to DEBUG and do like,
`LOG.debug("Bucket Layout: {}", bucketLayout);`
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
##########
@@ -78,6 +80,29 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
Preconditions.checkNotNull(allocateBlockRequest);
KeyArgs keyArgs = allocateBlockRequest.getKeyArgs();
+ String keyPath = keyArgs.getKeyName();
Review comment:
This functionality has duplicated over all the requests. Can you move this to OmClientRequest.java
`String OmClientRequest#validateAndNormalizeKey(BucketLayout bucketLayout, String keyPath, OzoneManager ozoneManager) throws OMException `
and we can use this method in the OM request classes.
`String keyPath = validateAndNormalizeKey(getBucketLayout(), keyArgs.getKeyName(), ozoneManager);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 closed pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
aryangupta1998 closed pull request #2819:
URL: https://github.com/apache/ozone/pull/2819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr merged pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr merged pull request #2819:
URL: https://github.com/apache/ozone/pull/2819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#issuecomment-965925213
@aryangupta1998 Good Work!
Overall the latest patch looks good. Just a comment, can you please update [KeyManager#lookupKey()](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L651) as well with the new validate method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on pull request #2819: HDDS-5929. Make FSO and OBS bucket layouts independent of normalization config flag
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on pull request #2819:
URL: https://github.com/apache/ozone/pull/2819#issuecomment-965977266
> @aryangupta1998 Good Work! Overall the latest patch looks good. Just a comment, can you please update [KeyManager#lookupKey()](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L651) as well with the new validate method.
Thanks, @rakeshadr for the reviews. I have addressed the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org