You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "fmorg-git (via GitHub)" <gi...@apache.org> on 2026/05/06 19:07:32 UTC
[PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
fmorg-git opened a new pull request, #10203:
URL: https://github.com/apache/ozone/pull/10203
Please describe your PR in detail:
* In STS, CopyObject requires s3:GetObject action on source and s3:PutObject action on destination in order to successfully invoke the api. In the latent S3 implementation, the code was doing an additional read on the destination object to get the eTag and modificationTime when the file was created. This is problematic, because AWS doesn't require s3:GetObject action on the destination file, so this extra read was causing STS smoke tests to fail.
In order to address, don't do the extra read (which will help performance as well) and instead make the commit key call return the modification time in the response and use that. Furthermore, the code already knows what the md5Hash/eTag is, so there is no need to read the destination object.
For consistency, make uploadPartCopy similarly return the modification time as well.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15182
## How was this patch tested?
unit tests, smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418117604
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
It looks like that LastModified element is a mandatory part of CopyObject and UploadPartCopy.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
So without this patch, Ozone returns incorrect LastModified? Since there is already lastModified field in CopyObjectResponse and CopyPartResult.
Since it's mandatory for S3, we don't need to maintain both commitKeyWithModificationTime and commitKey, update commitKey to return long, instead of OptionalLong, making sure lastModified is returned after a successful commit, keeping a clean logic and code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3460075040
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java:
##########
@@ -254,8 +255,9 @@ void commitKey(long offset) throws IOException {
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
+ modificationTime = commitUploadPartInfo.getModificationTime();
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ modificationTime = omClient.commitKey(buildKeyArgs(), openID);
Review Comment:
If it never reach here (e.g. uploading process failed), then would `modificationTime` equals zero? In that case, would it be better to use `Long` and check for nullity? If the modificationTime is only accessed after commit (seems to be the case), then I'm OK to leave it be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4644323078
commenting to remove stale label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418117604
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
It looks like that LastModified element is a mandatory part of CopyObject and UploadPartCopy.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
So without this patch, Ozone returns incorrect LastModified? Since there is already lastModified field in CopyObjectResponse and CopyPartResult.
Since it's mandatory for S3, we don't need to maintain both commitKeyWithModificationTime and commitKey, update commitKey to return long, instead of OptionalLong, keeping a clean code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3463386485
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java:
##########
@@ -254,8 +255,9 @@ void commitKey(long offset) throws IOException {
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
+ modificationTime = commitUploadPartInfo.getModificationTime();
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ modificationTime = omClient.commitKey(buildKeyArgs(), openID);
Review Comment:
yes, that's good observation that `modificationTime` would be 0 if the commit fails, and it is correct that `modificationTime` is only accessed after the commit is successful. One disadvantage would using `Long` is autoboxing and generating unnecessary `Long` objects, so I'm ok to leave as is (and I think it sounds like you are as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4640896431
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3425089191
##########
hadoop-ozone/interface-client/src/main/resources/proto.lock:
##########
@@ -5803,7 +5803,15 @@
]
},
{
- "name": "CommitKeyResponse"
+ "name": "CommitKeyResponse",
+ "fields": [
Review Comment:
updated - cd3d5b295c8a5e5fd8ad5f335f25f959db37be7a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4724855116
> > In the latent S3 implementation, the code was doing an additional read on the destination object to get the eTag and modificationTime when the file was created.
>
> Fabian, Can you point to the code which is doing this?
>
> > This is problematic, because AWS doesn't require s3:GetObject action on the destination file, so this extra read was causing STS smoke tests to fail
>
> Can you point out failed smoke test?
I removed the code that was doing the additional read on the destination object. Copilot flagged it as an issue in the previous PR as well but had an incorrect suggested fix: https://github.com/apache/ozone/pull/10197#discussion_r3386827556.
This series of smoke tests will fail without this fix: https://github.com/fmorg-git/ozone/blob/0ece11555841b68822e0f5a87c77fc4830ec4fda/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-sts.robot#L927-L984
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4715049805
@Fabian, could you also add some tests for the new changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418117604
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
It looks like that LastModified element is a mandatory part of CopyObject and UploadPartCopy.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
So without this patch, Ozone returns incorrect LastModified? Since there is already lastModified field in CopyObjectResponse and CopyPartResult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed to find what that modification time is. This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3432263201
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -1146,7 +1158,6 @@ default CancelPrepareResponse cancelOzoneManagerPrepare() throws IOException {
EchoRPCResponse echoRPCReq(byte[] payloadReq, int payloadSizeResp,
boolean writeToRatis) throws IOException;
-
Review Comment:
updated - 3a5e0256443495b969a6cf34b054c9546fbce928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4690127957
hi @ChenSammi - this PR has been rebased and ready to review once the workflow checks pass. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418162967
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntryPool.java:
##########
@@ -330,7 +332,10 @@ void commitKey(long offset) throws IOException {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ final OptionalLong modificationTime = omClient.commitKeyWithModificationTime(buildKeyArgs(), openID);
+ if (modificationTime.isPresent()) {
+ metadata.put(OzoneConsts.MODIFICATION_TIME, String.valueOf(modificationTime.getAsLong()));
Review Comment:
metadata is to persist in OM DB, a different goal. Please add a new field for LastModified value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418137879
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -1146,7 +1158,6 @@ default CancelPrepareResponse cancelOzoneManagerPrepare() throws IOException {
EchoRPCResponse echoRPCReq(byte[] payloadReq, int payloadSizeResp,
boolean writeToRatis) throws IOException;
-
Review Comment:
This can be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
my understanding is now the current time is returned, not necessarily the exact last modified time that was set when the key was committed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
my understanding is that now the current time is set when the key is committed. Then an extra lookup is performed to find what that modification time is. This patch changes to return the modification time in the CommitKeyResponse itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4714697418
> In the latent S3 implementation, the code was doing an additional read on the destination object to get the eTag and modificationTime when the file was created.
Can you point to the code which is doing this?
>
This is problematic, because AWS doesn't require s3:GetObject action on the destination file, so this extra read was causing STS smoke tests to fail
Can you point out failed smoke test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3460075040
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java:
##########
@@ -254,8 +255,9 @@ void commitKey(long offset) throws IOException {
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
+ modificationTime = commitUploadPartInfo.getModificationTime();
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ modificationTime = omClient.commitKey(buildKeyArgs(), openID);
Review Comment:
If it never reach here (e.g. uploading process failed), then would `modificationTime` equals zero? In that case, would it be better to use `Long` and check for nullity? If the modificationTime is only accessed after commit, then I'm OK to leave it be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3463386485
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java:
##########
@@ -254,8 +255,9 @@ void commitKey(long offset) throws IOException {
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
+ modificationTime = commitUploadPartInfo.getModificationTime();
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ modificationTime = omClient.commitKey(buildKeyArgs(), openID);
Review Comment:
yes, that's good observation that `modificationTime` would be 0 if the commit fails, and it is correct that `modificationTime` is only accessed after the commit is successful. One disadvantage with using `Long` is autoboxing and generating unnecessary `Long` objects, so I'm ok to leave as is (and I think it sounds like you are as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3463397630
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -985,8 +1002,19 @@ private Response createMultipartKey(OzoneVolume volume, OzoneBucket ozoneBucket,
}
}
+ private static OMException findOMException(Throwable t) {
+ Throwable currentException = t;
+ while (currentException != null) {
+ if (currentException instanceof OMException) {
+ return (OMException) currentException;
+ }
+ currentException = currentException.getCause();
+ }
+ return null;
Review Comment:
updated - 1ce9f3afa1aeac9d3318619a5ce5807b8624ce54
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3460044149
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -995,29 +1023,36 @@ void copy(OzoneVolume volume, DigestInputStream src, long srcKeyLen,
S3ConditionalRequest.WriteConditions writeConditions)
throws IOException {
long copyLength;
-
+ final String eTag;
+ final long modificationTime;
if (isDatastreamEnabled() && !(replication != null &&
replication.getReplicationType() == EC) &&
srcKeyLen > getDatastreamMinLength()) {
perf.appendStreamMode();
- copyLength = ObjectEndpointStreaming
+ final CopyResult copyResult = ObjectEndpointStreaming
.copyKeyWithStream(volume.getBucket(destBucket), destKey, srcKeyLen,
getChunkSize(), replication, metadata, src, perf, startNanos, tags,
writeConditions);
+ eTag = copyResult.getETag();
+ copyLength = copyResult.getSize();
+ modificationTime = copyResult.getModificationTime();
} else {
- try (OzoneOutputStream dest = openKeyForPut(
+ final OzoneOutputStream destStream = openKeyForPut(
volume.getName(), destBucket, destKey, srcKeyLen,
- replication, metadata, tags, writeConditions)) {
+ replication, metadata, tags, writeConditions);
+ try (OzoneOutputStream dest = destStream) {
long metadataLatencyNs =
getMetrics().updateCopyKeyMetadataStats(startNanos);
perf.appendMetaLatencyNanos(metadataLatencyNs);
copyLength = IOUtils.copyLarge(src, dest, 0, srcKeyLen, new byte[getIOBufferSize(srcKeyLen)]);
- final String md5Hash = DatatypeConverter.printHexBinary(src.getMessageDigest().digest()).toLowerCase();
- dest.getMetadata().put(OzoneConsts.ETAG, md5Hash);
+ eTag = DatatypeConverter.printHexBinary(src.getMessageDigest().digest()).toLowerCase();
+ destStream.getMetadata().put(OzoneConsts.ETAG, eTag);
Review Comment:
Nit: Unnecessary change, previously this is intentional since this means that the ETag is a md5Hash for this case (it might not be for other case).
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java:
##########
@@ -254,8 +255,9 @@ void commitKey(long offset) throws IOException {
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
+ modificationTime = commitUploadPartInfo.getModificationTime();
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ modificationTime = omClient.commitKey(buildKeyArgs(), openID);
Review Comment:
If it never reach here (e.g. uploading process failed), then would `modificationTime` equals zero? In that case, would it be better to use `Long` and check for nullity?
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -985,8 +1002,19 @@ private Response createMultipartKey(OzoneVolume volume, OzoneBucket ozoneBucket,
}
}
+ private static OMException findOMException(Throwable t) {
+ Throwable currentException = t;
+ while (currentException != null) {
+ if (currentException instanceof OMException) {
+ return (OMException) currentException;
+ }
+ currentException = currentException.getCause();
+ }
+ return null;
Review Comment:
Nit: Use the HddsClientUtils.containsException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
my understanding is that now the current time is set when the key is committed. Then an extra lookup is performed to find what that modification time is. This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3432107010
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntryPool.java:
##########
@@ -330,7 +332,10 @@ void commitKey(long offset) throws IOException {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(buildKeyArgs(), openID);
} else {
- omClient.commitKey(buildKeyArgs(), openID);
+ final OptionalLong modificationTime = omClient.commitKeyWithModificationTime(buildKeyArgs(), openID);
+ if (modificationTime.isPresent()) {
+ metadata.put(OzoneConsts.MODIFICATION_TIME, String.valueOf(modificationTime.getAsLong()));
Review Comment:
updated - 1f38ec45f28dfeb5f8d2fd798aa348d09d568844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3425931104
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> Since it's mandatory for S3, we don't need to maintain both commitKeyWithModificationTime and commitKey, update commitKey to return long,
@ChenSammi - would this be a breaking api change to commitKey for older clients or mixed client versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed to find what that modification time is (so it was precise). This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418149936
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CopyPartResult.java:
##########
@@ -46,8 +46,12 @@ public CopyPartResult() {
}
public CopyPartResult(String eTag) {
Review Comment:
Remove this constructor if it's not used anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed to find what that modification time is (so it was correct). This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed in the response itself so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed to find what that modification time is (so it was precise). This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed in the response itself so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3463345645
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java:
##########
@@ -995,29 +1023,36 @@ void copy(OzoneVolume volume, DigestInputStream src, long srcKeyLen,
S3ConditionalRequest.WriteConditions writeConditions)
throws IOException {
long copyLength;
-
+ final String eTag;
+ final long modificationTime;
if (isDatastreamEnabled() && !(replication != null &&
replication.getReplicationType() == EC) &&
srcKeyLen > getDatastreamMinLength()) {
perf.appendStreamMode();
- copyLength = ObjectEndpointStreaming
+ final CopyResult copyResult = ObjectEndpointStreaming
.copyKeyWithStream(volume.getBucket(destBucket), destKey, srcKeyLen,
getChunkSize(), replication, metadata, src, perf, startNanos, tags,
writeConditions);
+ eTag = copyResult.getETag();
+ copyLength = copyResult.getSize();
+ modificationTime = copyResult.getModificationTime();
} else {
- try (OzoneOutputStream dest = openKeyForPut(
+ final OzoneOutputStream destStream = openKeyForPut(
volume.getName(), destBucket, destKey, srcKeyLen,
- replication, metadata, tags, writeConditions)) {
+ replication, metadata, tags, writeConditions);
+ try (OzoneOutputStream dest = destStream) {
long metadataLatencyNs =
getMetrics().updateCopyKeyMetadataStats(startNanos);
perf.appendMetaLatencyNanos(metadataLatencyNs);
copyLength = IOUtils.copyLarge(src, dest, 0, srcKeyLen, new byte[getIOBufferSize(srcKeyLen)]);
- final String md5Hash = DatatypeConverter.printHexBinary(src.getMessageDigest().digest()).toLowerCase();
- dest.getMetadata().put(OzoneConsts.ETAG, md5Hash);
+ eTag = DatatypeConverter.printHexBinary(src.getMessageDigest().digest()).toLowerCase();
+ destStream.getMetadata().put(OzoneConsts.ETAG, eTag);
Review Comment:
the setting of the `eTag` variable is needed lower down for the return value: `return new CopyResult(eTag, copyLength, modificationTime);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4787300877
Thanks @ivandika3 @adoroszlai for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418153118
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartCommitUploadPartInfo.java:
##########
@@ -27,9 +29,16 @@ public class OmMultipartCommitUploadPartInfo {
private final String eTag;
+ private final Long modificationTime;
+
public OmMultipartCommitUploadPartInfo(String partName, String eTag) {
+ this(partName, eTag, null);
Review Comment:
Remove this constructor if it's not used anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3432100081
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartCommitUploadPartInfo.java:
##########
@@ -27,9 +29,16 @@ public class OmMultipartCommitUploadPartInfo {
private final String eTag;
+ private final Long modificationTime;
+
public OmMultipartCommitUploadPartInfo(String partName, String eTag) {
+ this(partName, eTag, null);
Review Comment:
updated - 1f38ec45f28dfeb5f8d2fd798aa348d09d568844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed to find what that modification time is (so it was correct). This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key, so it was not correct but was approximate
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed in the response itself so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3432263840
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CopyPartResult.java:
##########
@@ -46,8 +46,12 @@ public CopyPartResult() {
}
public CopyPartResult(String eTag) {
Review Comment:
updated - 3a5e0256443495b969a6cf34b054c9546fbce928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418223155
##########
hadoop-ozone/interface-client/src/main/resources/proto.lock:
##########
@@ -5803,7 +5803,15 @@
]
},
{
- "name": "CommitKeyResponse"
+ "name": "CommitKeyResponse",
+ "fields": [
Review Comment:
> proto.lock change doesn't need be included in patch. It will be updated as a whole during Ozone release.
Small, but important correction: `proto.lock` **shouldn't** be changed (in regular PRs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4784127425
> Thanks @fmorg-git , mostly LGTM, could you add a S3 integration test (i.e. AbstractS3SDKV1Tests and AbstractS3SDKV2Tests).
>
> 1. Upload S3 object (both using normal PutObject and MPU)
> 2. Check that the modificationTime returned in the S3 response is the same as the modificationTime in OM DB
updated to add integration tests - 33525f74284bf42c81dbdf760999d1773a42c61e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4791686220
Thanks all for the review and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. Avoid extra read for modification time on CopyObject/CopyPart [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #10203:
URL: https://github.com/apache/ozone/pull/10203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418117604
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
It looks like that LastModified element is a mandatory part of CopyObject and UploadPartCopy.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
So without this patch, Ozone now returns incorrect LastModified? Since there is already lastModified field in CopyObjectResponse and CopyPartResult.
Since it's mandatory for S3, we don't need to maintain both commitKeyWithModificationTime and commitKey, update commitKey to return long, instead of OptionalLong, making sure lastModified is returned after a successful commit, so we can get a clean logic and code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418117604
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
It looks like that LastModified element is a mandatory part of CopyObject and UploadPartCopy.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
So without this patch, Ozone returns incorrect LastModified? Since there is already lastModified field in CopyObjectResponse and CopyPartResult.
Since it's mandatory for S3, we don't need to maintain both commitKeyWithModificationTime and commitKey, update commitKey to return long, instead of OptionalLong, making sure lastModified is returned after a successful commit, so we can get a clean logic and code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3418141094
##########
hadoop-ozone/interface-client/src/main/resources/proto.lock:
##########
@@ -5803,7 +5803,15 @@
]
},
{
- "name": "CommitKeyResponse"
+ "name": "CommitKeyResponse",
+ "fields": [
Review Comment:
proto.lock change doesn't need be included in patch. It will be updated as a whole during Ozone release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3424898175
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
> So without this patch, Ozone now returns incorrect LastModified?
There were two cases previously per my understanding:
1) CopyObject: the current time was set when the key is committed. Then an extra lookup is performed in `getClientProtocol().getKeyDetails()` to find what that modification time is (so it was correct). This patch changes to return the modification time in the CommitKeyResponse itself so the extra lookup isn't needed (and this extra lookup was breaking STS).
2) UploadPartCopy: it was just doing an `Instant.now()` in the `CopyPartResult` constructor, so the returned LastModified was the gateway's wall‑clock at response time, not the OM‑side commit time of the part key, so it was not correct but was approximate
For both these cases, the patch now returns the actual lastModified that was set at the time the key was committed in the response itself so that a) no extra fetch is needed and b) it is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on code in PR #10203:
URL: https://github.com/apache/ozone/pull/10203#discussion_r3432099272
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -252,6 +253,17 @@ default void commitKey(OmKeyArgs args, long clientID)
"this to be implemented, as write requests use a new approach.");
}
+ /**
+ * Commit a key and optionally return the stored modification time (epoch millis).
+ * <p>
+ * This is backward compatible with older OM versions which do not return
+ * the value in {@code CommitKeyResponse}.
+ */
+ default OptionalLong commitKeyWithModificationTime(OmKeyArgs args, long clientID) throws IOException {
Review Comment:
updated the api signature - 1f38ec45f28dfeb5f8d2fd798aa348d09d568844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15182. [STS] Remove latent extra read on CopyObject api - make CopyPart work similarly [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #10203:
URL: https://github.com/apache/ozone/pull/10203#issuecomment-4736497999
> @fabian, could you also add some tests for the new changes?
updated to add unit tests - 485a19ac0411ecf7083d23e25adebd8ecbfdb635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org