You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/29 21:44:50 UTC
[GitHub] [ozone] DaveTeng0 opened a new pull request, #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
DaveTeng0 opened a new pull request, #3728:
URL: https://github.com/apache/ozone/pull/3728
## What changes were proposed in this pull request?
Handle user defined metadata supplied through the s3 interface.
Reference - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingObjects.html
(This PR is a fork from https://github.com/apache/ozone/pull/3200, with updates from latest master branch and resolve conflicts if there is any.)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6440
## How was this patch tested?
Manually tested put, head and copy object.
Added acceptance tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r959816981
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java:
##########
@@ -240,6 +252,46 @@ private Iterator<? extends OzoneBucket> iterateBuckets(
}
}
+ protected Map<String, String> getCustomMetadataFromHeaders(
+ MultivaluedMap<String, String> requestHeaders) {
+ Map<String, String> customMetadata = new HashMap<>();
+ if (requestHeaders == null || requestHeaders.isEmpty()) {
+ return customMetadata;
+ }
+
+ Set<String> customMetadataKeys = requestHeaders.keySet().stream()
+ .filter(k -> k.startsWith(CUSTOM_METADATA_HEADER_PREFIX))
+ .collect(Collectors.toSet());
+ long sizeInBytes = 0;
+ if (!customMetadataKeys.isEmpty()) {
+ for (String key : customMetadataKeys) {
+ String mapKey =
+ key.substring(CUSTOM_METADATA_HEADER_PREFIX.length());
+ List<String> values = requestHeaders.get(key);
+ String value = StringUtils.join(values, ",");
+ sizeInBytes += mapKey.getBytes(UTF_8).length;
+ sizeInBytes += value.getBytes(UTF_8).length;
+ if (sizeInBytes > 2 * KB) {
Review Comment:
This value in the original PR is coming from S3's limit. It would be good to make it to `OzoneConsts.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3728:
URL: https://github.com/apache/ozone/pull/3728#issuecomment-1230895957
cc. [kerneltime](https://github.com/kerneltime), [adoroszlai](https://github.com/adoroszlai)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
neils-dev commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r978156777
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -67,14 +70,9 @@ public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationType type,
int replicationFactor) {
- this.volumeName = volumeName;
- this.bucketName = bucketName;
- this.name = keyName;
- this.dataSize = size;
- this.creationTime = Instant.ofEpochMilli(creationTime);
- this.modificationTime = Instant.ofEpochMilli(modificationTime);
- this.replicationConfig = ReplicationConfig.fromTypeAndFactor(type,
- ReplicationFactor.valueOf(replicationFactor));
+ this(volumeName, bucketName, keyName, size, creationTime, modificationTime,
+ ReplicationConfig.fromTypeAndFactor(type,
Review Comment:
Setting the instance `this(creationTime, modificationTime ) `can remain initialized with `Instant.ofEpochMilli(creationTime)` and `Instant.ofEpochMilli(modificationTime) `respectively.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKeyDetails.java:
##########
@@ -58,8 +56,8 @@ public OzoneKeyDetails(String volumeName, String bucketName, String keyName,
super(volumeName, bucketName, keyName, size, creationTime,
modificationTime, type, replicationFactor);
this.ozoneKeyLocations = ozoneKeyLocations;
- this.metadata = metadata;
this.feInfo = feInfo;
+ this.getMetadata().putAll(metadata);
Review Comment:
Thanks @DaveTeng0 for this important patch to handle user defined metadata supplied through the s3 interface.
Setting the metadata member of the `OzoneKey` class that` OzoneKeyDetails` is extended from should be done by a setter method or a protected member. Perhaps a setter method can be provided by the `OzoneKey` class to set the metadata member or the `OzoneKey` _private_ metadata member can be made accessible by changing to `protected`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r960286181
##########
hadoop-ozone/dist/src/main/smoketest/s3/objectputget.robot:
##########
@@ -159,3 +159,15 @@ Zero byte file
${result} = Execute AWSS3APICli and checkrc get-object --bucket ${BUCKET} --key ${PREFIX}/putobject/key=value/zerobyte --range bytes=0-10000 /tmp/testfile2.result 255
Should contain ${result} InvalidRange
+
+Create file with user defined metadata
+ Execute echo "Randomtext" > /tmp/testfile2
+ Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/putobject/custom-metadata/key1 --body /tmp/testfile2 --metadata="custom-key1=custom-value1,custom-key2=custom-value2"
+
+ ${result} = Execute AWSS3APICli head-object --bucket ${BUCKET} --key ${PREFIX}/putobject/custom-metadata/key1
+ Should contain ${result} \"custom-key1\": \"custom-value1\"
+ Should contain ${result} \"custom-key2\": \"custom-value2\"
+
+ ${result} = Execute ozone sh key info /s3v/${BUCKET}/${PREFIX}/putobject/custom-metadata/key1
+ Should contain ${result} \"custom-key1\" : \"custom-value1\"
+ Should contain ${result} \"custom-key2\" : \"custom-value2\"
Review Comment:
Can you add a test for skipping GDPR and exceeding size limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r960285925
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -391,6 +391,8 @@ private OzoneConsts() {
public static final int S3_SECRET_KEY_MIN_LENGTH = 8;
+ public static final int S3_REQUEST_HEADER_METADATA_SIZE_LIMIT = 2;
Review Comment:
```suggestion
public static final int S3_REQUEST_HEADER_METADATA_SIZE_LIMIT_KB = 2;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r977016390
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java:
##########
@@ -240,6 +257,57 @@ private Iterator<? extends OzoneBucket> iterateBuckets(
}
}
+ protected Map<String, String> getCustomMetadataFromHeaders(
Review Comment:
It might be good to write a unit test for this to make sure the filtering is working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r977040296
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java:
##########
@@ -240,6 +257,57 @@ private Iterator<? extends OzoneBucket> iterateBuckets(
}
}
+ protected Map<String, String> getCustomMetadataFromHeaders(
Review Comment:
sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r979070789
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -67,14 +70,9 @@ public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationType type,
int replicationFactor) {
- this.volumeName = volumeName;
- this.bucketName = bucketName;
- this.name = keyName;
- this.dataSize = size;
- this.creationTime = Instant.ofEpochMilli(creationTime);
- this.modificationTime = Instant.ofEpochMilli(modificationTime);
- this.replicationConfig = ReplicationConfig.fromTypeAndFactor(type,
- ReplicationFactor.valueOf(replicationFactor));
+ this(volumeName, bucketName, keyName, size, creationTime, modificationTime,
+ ReplicationConfig.fromTypeAndFactor(type,
Review Comment:
The constructor called internally does the same, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r959821372
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -57,6 +64,11 @@ public class OzoneKey {
private ReplicationConfig replicationConfig;
+ private Map<String, String> metadata = new HashMap<>();
+
+ private Set<String> excludeMetadataFields =
+ new HashSet<>(Arrays.asList("gdprEnabled"));
Review Comment:
Pick the string up from `OzoneConsts`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r959820591
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -210,6 +210,13 @@ public Response put(
"Connection", "close").build();
}
+ // Normal put object
+ bucket = getBucket(bucketName);
+ Map<String, String> customMetadata =
+ getCustomMetadataFromHeaders(headers.getRequestHeaders());
Review Comment:
Filter the metadata here instead of filtering it in Key creation. Filtering it in Key construction might have unintended consequences if the key is supposed to have one of the reserved metadata keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r959850430
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java:
##########
@@ -240,6 +252,46 @@ private Iterator<? extends OzoneBucket> iterateBuckets(
}
}
+ protected Map<String, String> getCustomMetadataFromHeaders(
+ MultivaluedMap<String, String> requestHeaders) {
+ Map<String, String> customMetadata = new HashMap<>();
+ if (requestHeaders == null || requestHeaders.isEmpty()) {
+ return customMetadata;
+ }
+
+ Set<String> customMetadataKeys = requestHeaders.keySet().stream()
+ .filter(k -> k.startsWith(CUSTOM_METADATA_HEADER_PREFIX))
+ .collect(Collectors.toSet());
+ long sizeInBytes = 0;
+ if (!customMetadataKeys.isEmpty()) {
+ for (String key : customMetadataKeys) {
+ String mapKey =
+ key.substring(CUSTOM_METADATA_HEADER_PREFIX.length());
+ List<String> values = requestHeaders.get(key);
+ String value = StringUtils.join(values, ",");
+ sizeInBytes += mapKey.getBytes(UTF_8).length;
+ sizeInBytes += value.getBytes(UTF_8).length;
+ if (sizeInBytes > 2 * KB) {
Review Comment:
make sense! Sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r978167396
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKeyDetails.java:
##########
@@ -58,8 +56,8 @@ public OzoneKeyDetails(String volumeName, String bucketName, String keyName,
super(volumeName, bucketName, keyName, size, creationTime,
modificationTime, type, replicationFactor);
this.ozoneKeyLocations = ozoneKeyLocations;
- this.metadata = metadata;
this.feInfo = feInfo;
+ this.getMetadata().putAll(metadata);
Review Comment:
oh!! good catch! let me add a setter method for it! Thanks Neils!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r960843899
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -391,6 +391,8 @@ private OzoneConsts() {
public static final int S3_SECRET_KEY_MIN_LENGTH = 8;
+ public static final int S3_REQUEST_HEADER_METADATA_SIZE_LIMIT = 2;
Review Comment:
make sense!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime merged pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime merged PR #3728:
URL: https://github.com/apache/ozone/pull/3728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r959859736
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -210,6 +210,13 @@ public Response put(
"Connection", "close").build();
}
+ // Normal put object
+ bucket = getBucket(bucketName);
+ Map<String, String> customMetadata =
+ getCustomMetadataFromHeaders(headers.getRequestHeaders());
Review Comment:
Make sense! Sure!!
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -57,6 +64,11 @@ public class OzoneKey {
private ReplicationConfig replicationConfig;
+ private Map<String, String> metadata = new HashMap<>();
+
+ private Set<String> excludeMetadataFields =
+ new HashSet<>(Arrays.asList("gdprEnabled"));
Review Comment:
sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r960843586
##########
hadoop-ozone/dist/src/main/smoketest/s3/objectputget.robot:
##########
@@ -159,3 +159,15 @@ Zero byte file
${result} = Execute AWSS3APICli and checkrc get-object --bucket ${BUCKET} --key ${PREFIX}/putobject/key=value/zerobyte --range bytes=0-10000 /tmp/testfile2.result 255
Should contain ${result} InvalidRange
+
+Create file with user defined metadata
+ Execute echo "Randomtext" > /tmp/testfile2
+ Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/putobject/custom-metadata/key1 --body /tmp/testfile2 --metadata="custom-key1=custom-value1,custom-key2=custom-value2"
+
+ ${result} = Execute AWSS3APICli head-object --bucket ${BUCKET} --key ${PREFIX}/putobject/custom-metadata/key1
+ Should contain ${result} \"custom-key1\": \"custom-value1\"
+ Should contain ${result} \"custom-key2\": \"custom-value2\"
+
+ ${result} = Execute ozone sh key info /s3v/${BUCKET}/${PREFIX}/putobject/custom-metadata/key1
+ Should contain ${result} \"custom-key1\" : \"custom-value1\"
+ Should contain ${result} \"custom-key2\" : \"custom-value2\"
Review Comment:
make sense! sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r978174782
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -67,14 +70,9 @@ public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationType type,
int replicationFactor) {
- this.volumeName = volumeName;
- this.bucketName = bucketName;
- this.name = keyName;
- this.dataSize = size;
- this.creationTime = Instant.ofEpochMilli(creationTime);
- this.modificationTime = Instant.ofEpochMilli(modificationTime);
- this.replicationConfig = ReplicationConfig.fromTypeAndFactor(type,
- ReplicationFactor.valueOf(replicationFactor));
+ this(volumeName, bucketName, keyName, size, creationTime, modificationTime,
+ ReplicationConfig.fromTypeAndFactor(type,
Review Comment:
@neils-dev can you please explain, I am not sure I understood your comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3728:
URL: https://github.com/apache/ozone/pull/3728#issuecomment-1247167263
cc @umamaheswararao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r980529637
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -67,14 +70,9 @@ public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationType type,
int replicationFactor) {
- this.volumeName = volumeName;
- this.bucketName = bucketName;
- this.name = keyName;
- this.dataSize = size;
- this.creationTime = Instant.ofEpochMilli(creationTime);
- this.modificationTime = Instant.ofEpochMilli(modificationTime);
- this.replicationConfig = ReplicationConfig.fromTypeAndFactor(type,
- ReplicationFactor.valueOf(replicationFactor));
+ this(volumeName, bucketName, keyName, size, creationTime, modificationTime,
+ ReplicationConfig.fromTypeAndFactor(type,
Review Comment:
Yes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on a diff in pull request #3728: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
neils-dev commented on code in PR #3728:
URL: https://github.com/apache/ozone/pull/3728#discussion_r978240463
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -67,14 +70,9 @@ public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationType type,
int replicationFactor) {
- this.volumeName = volumeName;
- this.bucketName = bucketName;
- this.name = keyName;
- this.dataSize = size;
- this.creationTime = Instant.ofEpochMilli(creationTime);
- this.modificationTime = Instant.ofEpochMilli(modificationTime);
- this.replicationConfig = ReplicationConfig.fromTypeAndFactor(type,
- ReplicationFactor.valueOf(replicationFactor));
+ this(volumeName, bucketName, keyName, size, creationTime, modificationTime,
+ ReplicationConfig.fromTypeAndFactor(type,
Review Comment:
Sure @kerneltime, the intialization for the creationTime, modification can retain its previous initialization prior to the change:
**_previous:_**
```
this.creationTime = Instant.ofEpochMilli(creationTime);
this.modificationTime = Instant.ofEpochMilli(modificationTime);
```
**_now:_**
line 73,
`this(volumeName, bucketName, keyName, size, `**Instant.ofEpochMilli(creationTime)**, **Instant.ofEpochMilli(modificationTime),**` ...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org