You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2023/12/18 07:23:47 UTC

[PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

sumitagrawl opened a new pull request, #5811:
URL: https://github.com/apache/ozone/pull/5811

   ## What changes were proposed in this pull request?
   - take snapshot should not over-write the db transaction with getLastAppliedIndex() which might be old
     As this was causing replay of transaction if db transaction is over-written on startup.
   This can get trigger from 
   1. db snapshot -- need wait for double buffer flush to complete on-going update
   2. upgrade prepare -- there is already a check with wait for update prepare index, so no need update back with LastAppliedTransactionId. Further code is simplified.
   
   - simplify updateLastAppliedIndex() call updated only using double buffer flush. This update is required for Leader state isReady() by ratis as present in comment.
   a. notifyTermIndexUpdated() should be updated using double buffer so that term index always in sync with db transaction info
   b. doubleBuffer flush simplify update the transaction index as present in applyTransactionMap or ratisTransactionMap
   
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-9342
   
   ## How was this patch tested?
   
   - existing test cases should cover impact
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5811:
URL: https://github.com/apache/ozone/pull/5811#issuecomment-1862186884

   > * simplify updateLastAppliedIndex() call updated only using double buffer flush.
   >   a. notifyTermIndexUpdated() should be updated using double buffer so that term index always in sync with db transaction info. This update is required for Leader state isReady() by ratis as present in comment.
   >   b. doubleBuffer flush simplify update the transaction index as present in applyTransactionMap or ratisTransactionMap
   
   ok, seperated, handled as part of HDDS-9960.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5811:
URL: https://github.com/apache/ozone/pull/5811#discussion_r1434826620


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -489,19 +489,24 @@ public OzoneManagerDoubleBuffer buildDoubleBufferForRatis() {
    */
   @Override
   public long takeSnapshot() throws IOException {
-    LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
+    try {
+      awaitDoubleBufferFlush();
+    } catch (InterruptedException e) {
+      LOG.warn("double buffer wait is interrupted, ", e);
+      throw IOUtils.toInterruptedIOException("takeSnapshot is interrupted", e);
+    }
+    ozoneManager.getMetadataManager().getStore().flushDB();
+    TransactionInfo transactionInfo = ozoneManager.getMetadataManager()
+        .getTransactionInfoTable().get(TRANSACTION_INFO_KEY);
     TermIndex lastTermIndex = getLastAppliedTermIndex();
-    long lastAppliedIndex = lastTermIndex.getIndex();
+    LOG.info("Current Transaction Index {} lastAppliedTermIndex {}",
+        transactionInfo, lastTermIndex);
+    // update with lastAppliedTermIndex as snapshot have dependency with
+    // ratis commit index as triggered during notifyTermIndexUpdated after
+    // apply transaction. Can not use TRANSACTION_INFO_KEY from db
     snapshotInfo.updateTermIndex(lastTermIndex.getTerm(),

Review Comment:
   > Should it be updated with `transactionInfo` instead? We should use the on disk state instead of the memory state for snapshots.
   
   @szetszwo 
   As discussed, I am updating index to double buffer using callback mechanism. Please have a look,
   https://github.com/apache/ozone/pull/5857
   
   We can discard this PR if above seems ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #5811:
URL: https://github.com/apache/ozone/pull/5811#discussion_r1434333956


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -489,19 +489,24 @@ public OzoneManagerDoubleBuffer buildDoubleBufferForRatis() {
    */
   @Override
   public long takeSnapshot() throws IOException {
-    LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
+    try {
+      awaitDoubleBufferFlush();
+    } catch (InterruptedException e) {
+      LOG.warn("double buffer wait is interrupted, ", e);
+      throw IOUtils.toInterruptedIOException("takeSnapshot is interrupted", e);
+    }
+    ozoneManager.getMetadataManager().getStore().flushDB();
+    TransactionInfo transactionInfo = ozoneManager.getMetadataManager()
+        .getTransactionInfoTable().get(TRANSACTION_INFO_KEY);
     TermIndex lastTermIndex = getLastAppliedTermIndex();
-    long lastAppliedIndex = lastTermIndex.getIndex();
+    LOG.info("Current Transaction Index {} lastAppliedTermIndex {}",
+        transactionInfo, lastTermIndex);
+    // update with lastAppliedTermIndex as snapshot have dependency with
+    // ratis commit index as triggered during notifyTermIndexUpdated after
+    // apply transaction. Can not use TRANSACTION_INFO_KEY from db
     snapshotInfo.updateTermIndex(lastTermIndex.getTerm(),

Review Comment:
   Should it be updated with `transactionInfo` instead?  We should use the on disk state instead of the memory state for snapshots.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5811:
URL: https://github.com/apache/ozone/pull/5811#issuecomment-1868342990

   If this can fix [HDDS-9342_reproduce_20231217.patch](https://issues.apache.org/jira/secure/attachment/13065365/HDDS-9342_reproduce_20231217.patch), it is okay to merge this as a workaround.  We will fix the underlying logic in separated JIRAs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5811:
URL: https://github.com/apache/ozone/pull/5811#issuecomment-1866445271

   > @sumitagrawl Thanks for working on this!
   > 
   > Could you test the change with [HDDS-9342_reproduce_20231217.patch](https://issues.apache.org/jira/secure/attachment/13065365/HDDS-9342_reproduce_20231217.patch)? If this PR cannot fix it, please file a different JIRA for this `takeSnapshot()` change. The pr for [HDDS-9342](https://issues.apache.org/jira/browse/HDDS-9342) must fix `HDDS-9342_reproduce_20231217.patch`.
   > 
   > BTW, there are many test failed with timeout. Please take a look.
   
   The testcase `HDDS-9342_reproduce_20231217.patch` is passed as tried locally.
   
   Many testcase was failing as returning the db TRANSACTION_INFO_KEY index do not include ratis commit index provided by raits notifyTermIndexUpdated. So now using lastAppliedTransaction only, but no db update for TRANSACTION_INFO_KEY with lastAppliedIndex.
   Now testcases are passing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl closed pull request #5811: HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID
URL: https://github.com/apache/ozone/pull/5811


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5811:
URL: https://github.com/apache/ozone/pull/5811#issuecomment-1861273681

   > simplify updateLastAppliedIndex()  ...
   
   @sumitagrawl , thanks for working on this!  Let's move the `updateLastAppliedIndex()` to a different JIRA since it is more complicated?  The `takeSnapshot()` change is relatively easy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5811:
URL: https://github.com/apache/ozone/pull/5811#issuecomment-1876510184

   this is getting handled with
   HDDS-10026. Remove applyTransactionMap and ratisTransactionMap from OzoneManagerStateMachine.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #5811:
URL: https://github.com/apache/ozone/pull/5811#discussion_r1431948192


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -489,19 +488,24 @@ public OzoneManagerDoubleBuffer buildDoubleBufferForRatis() {
    */
   @Override
   public long takeSnapshot() throws IOException {
-    LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
-    TermIndex lastTermIndex = getLastAppliedTermIndex();
-    long lastAppliedIndex = lastTermIndex.getIndex();
-    snapshotInfo.updateTermIndex(lastTermIndex.getTerm(),
-        lastAppliedIndex);
-    TransactionInfo build = new TransactionInfo.Builder()
-        .setTransactionIndex(lastAppliedIndex)
-        .setCurrentTerm(lastTermIndex.getTerm()).build();
-    Table<String, TransactionInfo> txnInfoTable =
-        ozoneManager.getMetadataManager().getTransactionInfoTable();
-    txnInfoTable.put(TRANSACTION_INFO_KEY, build);
+    // when called from request prepare, there is a wait that
+    // double buffer is flushed. When called from db snapshot,
+    // need ensure flushed from double buffer
+    try {
+      awaitDoubleBufferFlush();
+    } catch (InterruptedException e) {
+      LOG.warn("double buffer wait is interrupted, ", e);
+      Thread.interrupted();

Review Comment:
   It should re-throw
   ```java
         throw IOUtils.toInterruptedIOException("takeSnapshot is interrupted", e);
   ```
   
   FYI, ` Thread.interrupted()` is incorrect.  It should be:
   ```java
         Thread.currentThread().interrupt();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9342. OM restart failed due to transactionLogIndex smaller than current updateID [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #5811:
URL: https://github.com/apache/ozone/pull/5811#discussion_r1435666532


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -489,19 +489,24 @@ public OzoneManagerDoubleBuffer buildDoubleBufferForRatis() {
    */
   @Override
   public long takeSnapshot() throws IOException {
-    LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
+    try {
+      awaitDoubleBufferFlush();
+    } catch (InterruptedException e) {
+      LOG.warn("double buffer wait is interrupted, ", e);
+      throw IOUtils.toInterruptedIOException("takeSnapshot is interrupted", e);
+    }
+    ozoneManager.getMetadataManager().getStore().flushDB();
+    TransactionInfo transactionInfo = ozoneManager.getMetadataManager()
+        .getTransactionInfoTable().get(TRANSACTION_INFO_KEY);
     TermIndex lastTermIndex = getLastAppliedTermIndex();
-    long lastAppliedIndex = lastTermIndex.getIndex();
+    LOG.info("Current Transaction Index {} lastAppliedTermIndex {}",
+        transactionInfo, lastTermIndex);
+    // update with lastAppliedTermIndex as snapshot have dependency with
+    // ratis commit index as triggered during notifyTermIndexUpdated after
+    // apply transaction. Can not use TRANSACTION_INFO_KEY from db
     snapshotInfo.updateTermIndex(lastTermIndex.getTerm(),

Review Comment:
   Let's don't make the logic more complicated.  The direction is to remove both `applyTransactionMap` and `ratisTransactionMap` from the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org