You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/03/15 21:30:16 UTC
[GitHub] [ozone] avijayanhwx opened a new pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
avijayanhwx opened a new pull request #3200:
URL: https://github.com/apache/ozone/pull/3200
## What changes were proposed in this pull request?
Handle user defined metadata supplied through the s3 interface.
Reference - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingObjects.html
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6440
## How was this patch tested?
Manually tested put, head and copy object.
Added acceptance tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a change in pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on a change in pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#discussion_r827475830
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java
##########
@@ -76,6 +80,22 @@ public OzoneKey(String volumeName, String bucketName,
ReplicationFactor.valueOf(replicationFactor));
}
+ @SuppressWarnings("parameternumber")
+ public OzoneKey(String volumeName, String bucketName,
+ String keyName, long size, long creationTime,
+ long modificationTime, ReplicationType type,
+ int replicationFactor, Map<String, String> metadata) {
+ this.volumeName = volumeName;
+ this.bucketName = bucketName;
+ this.name = keyName;
+ this.dataSize = size;
+ this.creationTime = Instant.ofEpochMilli(creationTime);
Review comment:
Also, nesting the constructors would help reduce code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a change in pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on a change in pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#discussion_r827472864
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java
##########
@@ -190,6 +202,38 @@ public void deleteS3Bucket(String s3BucketName)
}
}
+ protected Map<String, String> getCustomMetadataFromHeaders(
Review comment:
We should apply limits
```
Within the PUT request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value. ```
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#issuecomment-1068539090
LGTM except for a couple of changes. I think limits are important here. Also, a minor nit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#discussion_r830912922
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java
##########
@@ -93,6 +113,20 @@ public OzoneKey(String volumeName, String bucketName,
this.replicationConfig = replicationConfig;
}
+ @SuppressWarnings("parameternumber")
+ public OzoneKey(String volumeName, String bucketName,
+ String keyName, long size, long creationTime,
+ long modificationTime, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) {
+ this.volumeName = volumeName;
+ this.bucketName = bucketName;
+ this.name = keyName;
+ this.dataSize = size;
+ this.creationTime = Instant.ofEpochMilli(creationTime);
+ this.modificationTime = Instant.ofEpochMilli(modificationTime);
+ this.replicationConfig = replicationConfig;
Review comment:
```suggestion
this(volumeName, bucketName, keyName, size, creationTime,
modificationTime, replicationConfig);
```
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java
##########
@@ -76,6 +80,22 @@ public OzoneKey(String volumeName, String bucketName,
ReplicationFactor.valueOf(replicationFactor));
}
+ @SuppressWarnings("parameternumber")
+ public OzoneKey(String volumeName, String bucketName,
+ String keyName, long size, long creationTime,
+ long modificationTime, ReplicationType type,
+ int replicationFactor, Map<String, String> metadata) {
Review comment:
I think we can omit this constructor (with type and factor), it should be deprecated in favor of `ReplicationConfig` right away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#issuecomment-1075589458
Thanks for the review, will address them!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a change in pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on a change in pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#discussion_r827473186
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java
##########
@@ -76,6 +80,22 @@ public OzoneKey(String volumeName, String bucketName,
ReplicationFactor.valueOf(replicationFactor));
}
+ @SuppressWarnings("parameternumber")
+ public OzoneKey(String volumeName, String bucketName,
+ String keyName, long size, long creationTime,
+ long modificationTime, ReplicationType type,
+ int replicationFactor, Map<String, String> metadata) {
+ this.volumeName = volumeName;
+ this.bucketName = bucketName;
+ this.name = keyName;
+ this.dataSize = size;
+ this.creationTime = Instant.ofEpochMilli(creationTime);
Review comment:
These should be set by OM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a change in pull request #3200: HDDS-6440. Handle custom metadata (x-amz-meta) during put-object through S3 API.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on a change in pull request #3200:
URL: https://github.com/apache/ozone/pull/3200#discussion_r827473186
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java
##########
@@ -76,6 +80,22 @@ public OzoneKey(String volumeName, String bucketName,
ReplicationFactor.valueOf(replicationFactor));
}
+ @SuppressWarnings("parameternumber")
+ public OzoneKey(String volumeName, String bucketName,
+ String keyName, long size, long creationTime,
+ long modificationTime, ReplicationType type,
+ int replicationFactor, Map<String, String> metadata) {
+ this.volumeName = volumeName;
+ this.bucketName = bucketName;
+ this.name = keyName;
+ this.dataSize = size;
+ this.creationTime = Instant.ofEpochMilli(creationTime);
Review comment:
This should be set by OM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org