You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/04/12 10:50:06 UTC
[GitHub] [ozone] JyotinderSingh opened a new pull request, #3296: HDDS-6561. Modify default bucket layout to LEGACY.
JyotinderSingh opened a new pull request, #3296:
URL: https://github.com/apache/ozone/pull/3296
## What changes were proposed in this pull request?
This task to modify the existing default bucket layout at the OM server and Ozone shell command to LEGACY.
[OMConfigKeys.java#L283](https://github.com/apache/ozone/blob/master/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java#L283)
```java
ozone.default.bucket.layout = LEGACY
```
[CreateBucketHandler.java#L64](https://github.com/apache/ozone/blob/master/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java#L64)
```java
@Option(names = { "--layout", "-l" },
description = "Allowed Bucket Layouts: ${COMPLETION-CANDIDATES}",
defaultValue = "OBJECT_STORE")
```
Changing to LEGACY will make the cluster upgrades easier. An older version client can talk to the new OM server using LEGACY buckets.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6561
## How was this patch tested?
Integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r848364267
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3043,18 +3043,18 @@
<property>
<name>ozone.default.bucket.layout</name>
- <value>OBJECT_STORE</value>
+ <value>LEGACY</value>
Review Comment:
How about keeping it empty and not include `LEGACY` into the supported values.
` <value/>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on PR #3296:
URL: https://github.com/apache/ozone/pull/3296#issuecomment-1096670334
Please remove the test config that introduced to satisfy default OBS behavior at the OM server side.
OzoneConfigKeys.java
```
public static final String OZONE_CLIENT_TEST_OFS_DEFAULT_BUCKET_LAYOUT =
"ozone.client.test.ofs.default.bucket.layout";
// Fetches the bucket layout to be used by OFS.
this.defaultOFSBucketLayout = BucketLayout.fromString(
conf.get(OzoneConfigKeys.OZONE_CLIENT_TEST_OFS_DEFAULT_BUCKET_LAYOUT,
OzoneConfigKeys.OZONE_CLIENT_TEST_OFS_BUCKET_LAYOUT_DEFAULT));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r855162905
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java:
##########
@@ -57,11 +57,11 @@ public class CreateBucketHandler extends BucketHandler {
" user if not specified")
private String ownerName;
- enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE }
+ enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY }
Review Comment:
I agree to be mentioned LEGACY as defaultValue, but is that necessary to include `LEGACY` to the allowed list ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r851916728
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3043,18 +3043,18 @@
<property>
<name>ozone.default.bucket.layout</name>
- <value>OBJECT_STORE</value>
+ <value>LEGACY</value>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r855163913
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java:
##########
@@ -57,11 +57,11 @@ public class CreateBucketHandler extends BucketHandler {
" user if not specified")
private String ownerName;
- enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE }
+ enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY }
Review Comment:
The method would throw an error if we do not include the default value in the allowed list. That's why I have included it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on PR #3296:
URL: https://github.com/apache/ozone/pull/3296#issuecomment-1101143938
> Please remove the test config that introduced to satisfy default OBS behavior at the OM server side.
>
> OzoneConfigKeys.java
>
> ```
>
> public static final String OZONE_CLIENT_TEST_OFS_DEFAULT_BUCKET_LAYOUT =
> "ozone.client.test.ofs.default.bucket.layout";
>
> // Fetches the bucket layout to be used by OFS.
> this.defaultOFSBucketLayout = BucketLayout.fromString(
> conf.get(OzoneConfigKeys.OZONE_CLIENT_TEST_OFS_DEFAULT_BUCKET_LAYOUT,
> OzoneConfigKeys.OZONE_CLIENT_TEST_OFS_BUCKET_LAYOUT_DEFAULT));
> ```
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r848295039
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestObjectStore.java:
##########
@@ -84,17 +84,17 @@ public void testCreateBucketWithBucketLayout() throws Exception {
store.createVolume(sampleVolumeName);
OzoneVolume volume = store.getVolume(sampleVolumeName);
- // Case 1: Bucket layout: Empty and OM default bucket layout: OBJECT_STORE
+ // Case 1: Bucket layout: Empty and OM default bucket layout: LEGACY
Review Comment:
By default, the buckets now get created in `LEGACY` bucket layout. Hence, the test modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on PR #3296:
URL: https://github.com/apache/ozone/pull/3296#issuecomment-1097786239
@JyotinderSingh Adding one more point to the above, no need to persist LEGACY layout in the OM DB. Only persist OBS or FSO layouts. Please feel free to create a separate task if it requires more changes.
```
public class OmBucketInfoCodec implements Codec<OmBucketInfo> {
@Override
public byte[] toPersistedFormat(OmBucketInfo object) throws IOException {
Preconditions
.checkNotNull(object, "Null object can't be converted to byte array.");
return object.getProtobuf().toByteArray();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r855163913
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java:
##########
@@ -57,11 +57,11 @@ public class CreateBucketHandler extends BucketHandler {
" user if not specified")
private String ownerName;
- enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE }
+ enum AllowedBucketLayouts { FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY }
Review Comment:
The method would throw an error if we do not include the default value in the allowed list. That's why Ibhave included it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr merged pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr merged PR #3296:
URL: https://github.com/apache/ozone/pull/3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r848295594
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java:
##########
@@ -963,20 +963,4 @@ public void testListVolumeBucketKeyShouldPrintValidJsonArray()
testVolumes.forEach(vol -> execute(ozoneShell, new String[] {
"volume", "delete", vol}));
}
-
- @Test
- public void testClientBucketLayoutValidation() throws Exception {
- String[] args = new String[]{
- "bucket", "create", "o3://" + omServiceId + "/volume7" + "/bucketTest1",
- "--layout", "LEGACY"
- };
- try {
- execute(ozoneShell, args);
- Assert.fail("Should throw exception on unsupported bucket layouts!");
- } catch (Exception e) {
- GenericTestUtils.assertExceptionContains(
- "expected one of [FILE_SYSTEM_OPTIMIZED, OBJECT_STORE] ",
- e);
- }
- }
Review Comment:
This test is no longer valid since `LEGACY` is now added to the list of valid bucket layouts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r849267199
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java:
##########
@@ -963,20 +963,4 @@ public void testListVolumeBucketKeyShouldPrintValidJsonArray()
testVolumes.forEach(vol -> execute(ozoneShell, new String[] {
"volume", "delete", vol}));
}
-
- @Test
- public void testClientBucketLayoutValidation() throws Exception {
- String[] args = new String[]{
- "bucket", "create", "o3://" + omServiceId + "/volume7" + "/bucketTest1",
- "--layout", "LEGACY"
- };
- try {
- execute(ozoneShell, args);
- Assert.fail("Should throw exception on unsupported bucket layouts!");
- } catch (Exception e) {
- GenericTestUtils.assertExceptionContains(
- "expected one of [FILE_SYSTEM_OPTIMIZED, OBJECT_STORE] ",
- e);
- }
- }
Review Comment:
@JyotinderSingh I'd like to not to expose LEGACY layout and internally defaulting to LEGACY if not given/passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3296: HDDS-6561. Modify default bucket layout to LEGACY.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3296:
URL: https://github.com/apache/ozone/pull/3296#discussion_r851917347
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java:
##########
@@ -963,20 +963,4 @@ public void testListVolumeBucketKeyShouldPrintValidJsonArray()
testVolumes.forEach(vol -> execute(ozoneShell, new String[] {
"volume", "delete", vol}));
}
-
- @Test
- public void testClientBucketLayoutValidation() throws Exception {
- String[] args = new String[]{
- "bucket", "create", "o3://" + omServiceId + "/volume7" + "/bucketTest1",
- "--layout", "LEGACY"
- };
- try {
- execute(ozoneShell, args);
- Assert.fail("Should throw exception on unsupported bucket layouts!");
- } catch (Exception e) {
- GenericTestUtils.assertExceptionContains(
- "expected one of [FILE_SYSTEM_OPTIMIZED, OBJECT_STORE] ",
- e);
- }
- }
Review Comment:
LEGACY is no longer listed as the default value for the default bucket layout config key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org