You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2024/01/22 05:09:00 UTC

[PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

smengcl opened a new pull request, #6046:
URL: https://github.com/apache/ozone/pull/6046

   ## What changes were proposed in this pull request?
   
   Currently, only those keys in `KeyTable` would have metadata `HSYNC_CLIENT_ID` when those keys have been hsync'ed (and not closed/committed yet). The problem with this is that it makes `getExpiredOpenKeys()` and `listOpenKeys()` (HDDS-8830) inefficient by forcing them to look up `KeyTable` while they could have just used `OpenKeyTable` solely to determine whether an open key is hsync'ed or not.
   
   ### Changes
   
   1. During key `hsync()`, persist metadata `HSYNC_CLIENT_ID` to `OmKeyInfo` in `OpenKeyTable` as well, in addition to `KeyTable`. Only write when the client ID changes so it doesn't lead to excessive writes. Ideally, only the first hsync() would lead to a `OpenKeyTable` write of that key.
   2. During key `close()`, remove `HSYNC_CLIENT_ID` from OpenKeyTable if any, so that `HSYNC_CLIENT_ID` isn't unnecessarily written to the final key (which is useless when the key is committed/finalized.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-10077
   
   ## How was this patch tested?
   
   - [x] Existing tests that cover the changes shall pass.
   - [ ] Add new UT to cover the intended behavior.
     - When a key is newly created/opened, neither OpenKeyTable nor KeyTable should have hsync metadata.
     - When the key is hsync'ed, both OpenKeyTable and KeyTable should have hsync metadata.
     - When the key is hsync'ed again, both OpenKeyTable and KeyTable should have hsync metadata.
     - When the key is closed, KeyTable entry should no longer have hsync metadata. Key should not exist in OpenKeyTable.
     - All three types of buckets (LEGACY/OBS/FSO) should behave the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1914110919

   > So far it looks like a test issue rather than the code issue and I have adjusted the test to make it pass. Pls check if this is the desired behavior @ChenSammi .
   
   @smengcl , thanks for the finding. The fix is good. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #6046:
URL: https://github.com/apache/ozone/pull/6046


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1916186611

   Thanks @sumitagrawl @ChenSammi @jojochuang for reviewing this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1904457484

   @ashishkumar50 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463293776


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -337,6 +346,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
         // So that this key can't be committed again.
         omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
             dbOpenKey, trxnLogIndex);
+
+        // Prevent hsync metadata from getting committed to the final key
+        omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
+      } else if (updateOpenKeyTable) {
+        // Both isHSync and updateOpenKeyTable are true, update OpenKeyTable
+        omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
+            dbOpenKey, omKeyInfo, trxnLogIndex);

Review Comment:
   You cannot use the same omKeyInfo for keyTable and openKeyTable. There is one case, that a file's all blocks are per-allocated. Then the openKeyInfo will have all these blocks info, while the keyInfo only has the hsynced blocks info. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463908084


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -337,6 +346,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
         // So that this key can't be committed again.
         omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
             dbOpenKey, trxnLogIndex);
+
+        // Prevent hsync metadata from getting committed to the final key
+        omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
+      } else if (updateOpenKeyTable) {
+        // Both isHSync and updateOpenKeyTable are true, update OpenKeyTable
+        omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
+            dbOpenKey, omKeyInfo, trxnLogIndex);

Review Comment:
   Good catch. Let me add the hsync metadata tag solely without touching any other properties then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1470273224


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -260,15 +260,22 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
               " metadata while recovery flag is not set in request", KEY_UNDER_LEASE_RECOVERY);
         }
       }
-      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
-          commitKeyArgs.getMetadataList()));
+
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
 
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();
+        }
       } else if (isRecovery) {
         omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
         omKeyInfo.getMetadata().remove(OzoneConsts.LEASE_RECOVERY);
       }
+
+      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+          commitKeyArgs.getMetadataList()));

Review Comment:
   Done. Also kept the original semantic for `LEASE_RECOVERY` by keeping `if (isRecovery)` check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1464535165


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -263,8 +263,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
       omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
           commitKeyArgs.getMetadataList()));
 
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
+
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {

Review Comment:
   newOpenKeyInfo is created after omKeyInfo updated with commit metadata args above. This might have additional change of openKeyInfo. We can avoid this updating other info.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequestWithFSO.java:
##########
@@ -173,8 +174,16 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
 
       omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
           commitKeyArgs.getMetadataList()));
+
+      final String clientIdString = String.valueOf(writerClientId);
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
+
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, String.valueOf(writerClientId));
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenFileKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();

Review Comment:
   newOpenKeyInfo is created after omKeyInfo updated with commit metadata args above. This might have additional change of openKeyInfo. We can avoid this updating other info.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1905375966

   > @smengcl Thanks for working over this, having few query,
   > 
   > 1. adding metadata while commit will not resolve missing hsync commit for first time till fsync not called by client. Do we have plan to resolve by adding hsync flag while open keyfile ?
   
   Which issue are you talking about? If it is another existing jira then it is not in the scope of this PR.
   
   > 2. Related to performance, this interface will not be called in normal path. additionally checking key existence for open key is very fast. Do this provide any measurable performance improvement?
   
   It's not merely checking key existence in KeyTable, it is extra key look up + extra DB value deserialization + metadata check. Deserialization the large value byte array that stores `OmKeyInfo` is probably more expensive than DB get(seek).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1470209326


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -260,15 +260,22 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
               " metadata while recovery flag is not set in request", KEY_UNDER_LEASE_RECOVERY);
         }
       }
-      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
-          commitKeyArgs.getMetadataList()));
+
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
 
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();

Review Comment:
   Good idea. Then the mod time in open key would indicate the first time the key is hsync'ed (since open key will only be updated on the first hsync).
   
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1912805801

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463887946


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1888,19 +1859,19 @@ public ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold,
           } else {
             // add hsync'ed keys
             final KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
-                .setVolumeName(info.getVolumeName())
-                .setBucketName(info.getBucketName())
-                .setKeyName(info.getKeyName())
-                .setDataSize(info.getDataSize());
-            java.util.Optional.ofNullable(info.getLatestVersionLocations())
+                .setVolumeName(openKeyInfo.getVolumeName())
+                .setBucketName(openKeyInfo.getBucketName())
+                .setKeyName(openKeyInfo.getKeyName())
+                .setDataSize(openKeyInfo.getDataSize());

Review Comment:
   @sumitagrawl @ChenSammi Got it. Will use `KeyTable` in this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463905295


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -263,10 +264,18 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
       omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
           commitKeyArgs.getMetadataList()));
 
+      // flag to indicate whether it is necessary to update the value (ozone key) in OpenKeyTable
+      boolean updateOpenKeyTable = false;
+
       if (isHSync) {
+        updateOpenKeyTable = !OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey);
         omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
       } else if (isRecovery) {
+        // Q for Sammi/Wei-Chiu: The following line won't do anything (before this PR) because HSYNC_CLIENT_ID was never
+        // added to OpenKeyTable right? Have I missed something?

Review Comment:
   Hmm. While you are right that `OMRecoverLeaseRequest` could return either `keyInfo` or `openKeyInfo` here:
   
   https://github.com/apache/ozone/blob/04b6aa5e75ff0434043177c54e626a829a156032/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java#L275-L276
   
   but it still wouldn't update `OpenKeyTable` with `keyInfo` from `KeyTable` right? i.e.:
   
   1. update `OpenKeyTable` cache
   
   https://github.com/apache/ozone/blob/04b6aa5e75ff0434043177c54e626a829a156032/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java#L239-L241
   
   2. update `OpenKeyTable`
   
   https://github.com/apache/ozone/blob/b16087aa6c2d39bf98314b7a2e6d4811131ac26d/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMRecoverLeaseResponse.java#L65-L69



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463282997


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -263,10 +264,18 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
       omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
           commitKeyArgs.getMetadataList()));
 
+      // flag to indicate whether it is necessary to update the value (ozone key) in OpenKeyTable
+      boolean updateOpenKeyTable = false;
+
       if (isHSync) {
+        updateOpenKeyTable = !OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey);
         omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
       } else if (isRecovery) {
+        // Q for Sammi/Wei-Chiu: The following line won't do anything (before this PR) because HSYNC_CLIENT_ID was never
+        // added to OpenKeyTable right? Have I missed something?

Review Comment:
   Not really. The OMRecoverLeaseRequest will return either keyInfo or openKeyInfo depending on which one has the lastest block information, so there is chance that commitKeyArgs will have HSYNC_CLIENT_ID metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463282997


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -263,10 +264,18 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
       omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
           commitKeyArgs.getMetadataList()));
 
+      // flag to indicate whether it is necessary to update the value (ozone key) in OpenKeyTable
+      boolean updateOpenKeyTable = false;
+
       if (isHSync) {
+        updateOpenKeyTable = !OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey);
         omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
       } else if (isRecovery) {
+        // Q for Sammi/Wei-Chiu: The following line won't do anything (before this PR) because HSYNC_CLIENT_ID was never
+        // added to OpenKeyTable right? Have I missed something?

Review Comment:
   Not really. The OMRecoverLeaseRequest will return either keyInfo or openKeyInfo depending on which one has the lastest block information, so that commitKeyArgs will either have HSYNC_CLIENT_ID metadata or LEASE_RECOVERY. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1469156198


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java:
##########
@@ -238,7 +236,7 @@ private RecoverLeaseResponse doWork(OzoneManager ozoneManager,
       openKeyInfo.setModificationTime(Time.now());
       // add to cache.
       omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
-          new CacheKey<>(dbOpenFileKey), CacheValue.get(transactionLogIndex, openKeyInfo));
+          dbOpenFileKey, openKeyInfo, transactionLogIndex);

Review Comment:
   I see it's usually use CacheKey and CacheValue to wrapper the data in most places to put data in Cache. Why we don't need it here? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #6046:
URL: https://github.com/apache/ozone/pull/6046#issuecomment-1911712810

   Regarding the [test failure](https://github.com/apache/ozone/actions/runs/7662179618/job/20886152096?pr=6046#step:6:221) with the last commit:
   
   ```
   Error:    TestLeaseRecovery.testGetCommittedBlockLengthTimeout:330->closeIgnoringKeyNotFound:98 expected: <KEY_NOT_FOUND> but was: <KEY_UNDER_LEASE_RECOVERY>
   ```
   
   After a few hours of debugging, I found the new test `testGetCommittedBlockLengthTimeout` (added in HDDS-10044) does NOT throw `KEY_NOT_FOUND` when `forceRecovery == false` during `stream.close()`, because the key still exists in `OpenKeyTable` when the stream is being closed even **without this PR's change**. (Note when `forceRecovery` is `true`, the open key is indeed gone. Thus will throw `KEY_NOT_FOUND`.)
   
   This specific test wasn't previously failing because `HSYNC_CLIENT_ID` was NEVER put to `OpenKeyTable` before this patch, as I have [detailed](https://github.com/apache/ozone/pull/6046#discussion_r1463905295) above already. And now with this PR, `KEY_UNDER_LEASE_RECOVERY` is thrown here:
   
   https://github.com/apache/ozone/blob/a15c9ae1b30267db5000bfa2a307e9b57e601f6d/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java#L256-L262
   
   So far it looks like a test issue rather than the code issue and I have adjusted the test to make it pass. Pls check if this is the desired behavior @ChenSammi .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463293776


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -337,6 +346,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
         // So that this key can't be committed again.
         omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
             dbOpenKey, trxnLogIndex);
+
+        // Prevent hsync metadata from getting committed to the final key
+        omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
+      } else if (updateOpenKeyTable) {
+        // Both isHSync and updateOpenKeyTable are true, update OpenKeyTable
+        omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
+            dbOpenKey, omKeyInfo, trxnLogIndex);

Review Comment:
   You cannot use the same omKeyInfo for keyTable and openKeyTable. There is one case, that a file's all blocks are per-allocated. Then the openKeyInfo will have all these blocks info, while the keyInfo only has the hsynced block info. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1463290552


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1888,19 +1859,19 @@ public ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold,
           } else {
             // add hsync'ed keys
             final KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
-                .setVolumeName(info.getVolumeName())
-                .setBucketName(info.getBucketName())
-                .setKeyName(info.getKeyName())
-                .setDataSize(info.getDataSize());
-            java.util.Optional.ofNullable(info.getLatestVersionLocations())
+                .setVolumeName(openKeyInfo.getVolumeName())
+                .setBucketName(openKeyInfo.getBucketName())
+                .setKeyName(openKeyInfo.getKeyName())
+                .setDataSize(openKeyInfo.getDataSize());

Review Comment:
   Here in some case, keyInfo should be used, in other case, openKeyInfo should be used to commit the key. I did this check in OMRecoverLeaseRequest#doWork too. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1469163671


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -260,15 +260,22 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
               " metadata while recovery flag is not set in request", KEY_UNDER_LEASE_RECOVERY);
         }
       }
-      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
-          commitKeyArgs.getMetadataList()));
+
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
 
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();
+        }
       } else if (isRecovery) {
         omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
         omKeyInfo.getMetadata().remove(OzoneConsts.LEASE_RECOVERY);
       }
+
+      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+          commitKeyArgs.getMetadataList()));

Review Comment:
   Or you can remove L272-275, and add remove both HSYNC_CLIENT_ID and LEASE_RECOVERY in L349. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1469159750


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java:
##########
@@ -238,7 +236,7 @@ private RecoverLeaseResponse doWork(OzoneManager ozoneManager,
       openKeyInfo.setModificationTime(Time.now());
       // add to cache.
       omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
-          new CacheKey<>(dbOpenFileKey), CacheValue.get(transactionLogIndex, openKeyInfo));
+          dbOpenFileKey, openKeyInfo, transactionLogIndex);

Review Comment:
   Okay, it's below one, actually does the same thing.  Is they any benefit use the following one instead of the current one? 
   
   ```
   default void addCacheEntry(KEY cacheKey, VALUE value, long epoch) 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1469150696


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -260,15 +260,22 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
               " metadata while recovery flag is not set in request", KEY_UNDER_LEASE_RECOVERY);
         }
       }
-      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
-          commitKeyArgs.getMetadataList()));
+
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
 
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();
+        }
       } else if (isRecovery) {
         omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
         omKeyInfo.getMetadata().remove(OzoneConsts.LEASE_RECOVERY);
       }
+
+      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+          commitKeyArgs.getMetadataList()));

Review Comment:
   Please move this metadata.putAll before if(isHsync),  I believe the commitKeyArgs will have HSYNC_CLIENT_ID and 
   LEASE_RECOVERY metadata, so if you put it here, then the final committed key will still have these two metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1470194151


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java:
##########
@@ -238,7 +236,7 @@ private RecoverLeaseResponse doWork(OzoneManager ozoneManager,
       openKeyInfo.setModificationTime(Time.now());
       // add to cache.
       omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
-          new CacheKey<>(dbOpenFileKey), CacheValue.get(transactionLogIndex, openKeyInfo));
+          dbOpenFileKey, openKeyInfo, transactionLogIndex);

Review Comment:
   Yes. It is the same. It just makes the expression cleaner. Nicholas added it in a previous jira. I have mentioned that in a previously PR (after its merge) as well:
   
   https://github.com/apache/ozone/pull/5847#discussion_r1461295873



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1469170548


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -260,15 +260,22 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
               " metadata while recovery flag is not set in request", KEY_UNDER_LEASE_RECOVERY);
         }
       }
-      omKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
-          commitKeyArgs.getMetadataList()));
+
+      // non-null indicates it is necessary to update the open key
+      OmKeyInfo newOpenKeyInfo = null;
 
       if (isHSync) {
-        omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+        if (!OmKeyHSyncUtil.isHSyncedPreviously(omKeyInfo, clientIdString, dbOpenKey)) {
+          omKeyInfo.getMetadata().put(OzoneConsts.HSYNC_CLIENT_ID, clientIdString);
+          newOpenKeyInfo = omKeyInfo.copyObject();

Review Comment:
   The modification time should be updated for newOpenKeyInfo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1464286709


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -337,6 +346,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
         // So that this key can't be committed again.
         omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
             dbOpenKey, trxnLogIndex);
+
+        // Prevent hsync metadata from getting committed to the final key
+        omKeyInfo.getMetadata().remove(OzoneConsts.HSYNC_CLIENT_ID);
+      } else if (updateOpenKeyTable) {
+        // Both isHSync and updateOpenKeyTable are true, update OpenKeyTable
+        omMetadataManager.getOpenKeyTable(getBucketLayout()).addCacheEntry(
+            dbOpenKey, omKeyInfo, trxnLogIndex);

Review Comment:
   done in 7bdb08312a11ca9c37114eee0840be9a96460a01



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1888,19 +1859,19 @@ public ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold,
           } else {
             // add hsync'ed keys
             final KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
-                .setVolumeName(info.getVolumeName())
-                .setBucketName(info.getBucketName())
-                .setKeyName(info.getKeyName())
-                .setDataSize(info.getDataSize());
-            java.util.Optional.ofNullable(info.getLatestVersionLocations())
+                .setVolumeName(openKeyInfo.getVolumeName())
+                .setBucketName(openKeyInfo.getBucketName())
+                .setKeyName(openKeyInfo.getKeyName())
+                .setDataSize(openKeyInfo.getDataSize());

Review Comment:
   done in 9ac270e4ffab432014fbf9b19efe7d8fd21646f1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1462772760


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1888,19 +1859,19 @@ public ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold,
           } else {
             // add hsync'ed keys
             final KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
-                .setVolumeName(info.getVolumeName())
-                .setBucketName(info.getBucketName())
-                .setKeyName(info.getKeyName())
-                .setDataSize(info.getDataSize());
-            java.util.Optional.ofNullable(info.getLatestVersionLocations())
+                .setVolumeName(openKeyInfo.getVolumeName())
+                .setBucketName(openKeyInfo.getBucketName())
+                .setKeyName(openKeyInfo.getKeyName())
+                .setDataSize(openKeyInfo.getDataSize());

Review Comment:
   info used to have last commited datasize, commited blocks, but openKeyInfo may not have that information correctly. So IMO, can not use openKeyInfo for commit purpose.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6046:
URL: https://github.com/apache/ozone/pull/6046#discussion_r1462808433


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1888,19 +1859,19 @@ public ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold,
           } else {
             // add hsync'ed keys
             final KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
-                .setVolumeName(info.getVolumeName())
-                .setBucketName(info.getBucketName())
-                .setKeyName(info.getKeyName())
-                .setDataSize(info.getDataSize());
-            java.util.Optional.ofNullable(info.getLatestVersionLocations())
+                .setVolumeName(openKeyInfo.getVolumeName())
+                .setBucketName(openKeyInfo.getBucketName())
+                .setKeyName(openKeyInfo.getKeyName())
+                .setDataSize(openKeyInfo.getDataSize());

Review Comment:
   Good catch. Indeed, since it only updates `OpenKeyTable` on the first hsync call it won't always have the latest size:
   
   https://github.com/apache/ozone/blob/HDDS-7593/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java#L272
   
   **If** accurate data size here is a must, either `OpenKeyTable` have to be updated on every single hsync (which I don't like), or `getExpiredOpenKeys` have to resort to `KeyTable` again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org