You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2026/10/05 08:58:54 UTC
[PR] HDDS-16666. Flush SCM transaction in memory during apply transaction [ozone]
sarvekshayr opened a new pull request, #11409:
URL: https://github.com/apache/ozone/pull/11409
## What changes were proposed in this pull request?
When an SCM follower restarts after being down for a long time, Ratis syncs the log to disk quickly, but `applyTransaction` lags far behind, so `commitIndex` can exceed `appliedIndex` by millions of entries. All the DB updates produced while applying that backlog accumulate in the single in-memory `BatchOperation` held by `SCMHADBTransactionBufferImpl`, causing heavy memory pressure.
This change adds a count-based flush on the apply path:
- New config `ozone.scm.ha.dbtransactionbuffer.flush.pending.limit` (default `10000`): the number of buffered DB operations after which the buffer is flushed. Values `<= 0` are invalid and fall back to the default with a warning.
- New `SCMHADBTransactionBuffer#flushIfPendingLimitReached()`
- `SCMStateMachine#applyTransaction` calls it right after `updateLatestTrxInfo(...)` and `updateLastAppliedTermIndex(...)`, still inside the `beginApplyingTransaction()`/`endApplyingTransaction()` window. Flushing after the index is recorded keeps the persisted `TransactionInfo` consistent with the committed batch. The existing interval-based flush is unchanged.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16666
## How was this patch tested?
Added unit tests in `TestSCMHADBTransactionBufferImpl` and `TestSCMStateMachine`.
Manually tested on a SCM HA docker cluster with the flush pending limit set to 5:
- Flush lines were observed on all three SCMs, including followers, whenever the buffered operation count reached the limit.
(Note: Changed log level to `INFO` for testing purpose)
- With one follower stopped, load was generated, and the follower restarted: it logged repeated flushes while replaying the backlog and then `Follower caught up with leader`.
```
scm3-1 | 2026-10-05 06:43:38,126 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1 | 2026-10-05 06:43:38,130 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1 | 2026-10-05 06:43:38,133 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1 | 2026-10-05 06:43:38,138 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1 | 2026-10-05 06:43:38,141 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16666. Flush SCM transaction in memory during apply transaction [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #11409:
URL: https://github.com/apache/ozone/pull/11409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16666. Flush SCM transaction in memory during apply transaction [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #11409:
URL: https://github.com/apache/ozone/pull/11409#issuecomment-6012492239
Thanks @sumitagrawl and @yandrey321 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16666. Flush SCM transaction in memory during apply transaction [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11409:
URL: https://github.com/apache/ozone/pull/11409#discussion_r4184250808
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java:
##########
@@ -49,4 +67,101 @@ public void testRatisEventsRecording() throws Exception {
metrics.unRegister();
}
+
+ @Test
+ public void testApplyTransactionFlushesAfterRecordingTransactionInfo() throws Exception {
+ SCMHADBTransactionBuffer buffer = mock(SCMHADBTransactionBuffer.class);
+ SCMStateMachine stateMachine = newStateMachine(buffer, succeedingInvoker());
+
+ CompletableFuture<Message> result = stateMachine.applyTransaction(newTransaction(7));
+
+ assertTrue(result.isDone() && !result.isCompletedExceptionally());
+ InOrder order = inOrder(buffer);
+ order.verify(buffer).beginApplyingTransaction();
+ order.verify(buffer).updateLatestTrxInfo(TransactionInfo.valueOf(TermIndex.valueOf(1, 7)));
+ order.verify(buffer).flushIfPendingLimitReached();
+ order.verify(buffer).endApplyingTransaction();
+ }
+
+ @Test
+ public void testFlushCheckedForEveryAppliedTransaction() throws Exception {
+ SCMHADBTransactionBuffer buffer = mock(SCMHADBTransactionBuffer.class);
+ SCMStateMachine stateMachine = newStateMachine(buffer, succeedingInvoker());
+
+ for (int i = 1; i <= 5; i++) {
+ stateMachine.applyTransaction(newTransaction(i));
+ }
+
+ verify(buffer, times(5)).flushIfPendingLimitReached();
+ verify(buffer, times(5)).endApplyingTransaction();
+ }
+
+ @Test
+ public void testFlushStillCheckedWhenTransactionIsLogicallyRejected() throws Exception {
+ SCMHADBTransactionBuffer buffer = mock(SCMHADBTransactionBuffer.class);
+ ScmInvoker<?> invoker = mock(ScmInvoker.class);
+ when(invoker.invokeLocal(anyString(), any()))
+ .thenThrow(new SCMException("rejected", ResultCodes.FAILED_TO_FIND_CONTAINER));
+ SCMStateMachine stateMachine = newStateMachine(buffer, invoker);
+
+ CompletableFuture<Message> result = stateMachine.applyTransaction(newTransaction(3));
+
+ assertTrue(result.isCompletedExceptionally());
+ InOrder order = inOrder(buffer);
+ order.verify(buffer).updateLatestTrxInfo(TransactionInfo.valueOf(TermIndex.valueOf(1, 3)));
+ order.verify(buffer).flushIfPendingLimitReached();
+ order.verify(buffer).endApplyingTransaction();
+ }
+
+ @Test
+ public void testFlushFailureTerminatesAndClosesApplyingWindow() throws Exception {
+ ExitUtils.disableSystemExit();
+ try {
+ SCMHADBTransactionBuffer buffer = mock(SCMHADBTransactionBuffer.class);
+ doThrow(new IllegalStateException("injected flush failure")).when(buffer).flushIfPendingLimitReached();
+ SCMStateMachine stateMachine = newStateMachine(buffer, succeedingInvoker());
+
+ assertThrows(ExitUtils.ExitException.class, () -> stateMachine.applyTransaction(newTransaction(9)));
+
+ verify(buffer).endApplyingTransaction();
+ } finally {
+ ExitUtils.clear();
+ }
+ }
+
+ private static TransactionContext newTransaction(long index) throws Exception {
+ SCMRatisRequest request = SCMRatisRequest.of(RequestType.PIPELINE, "op", new Class<?>[0]);
+ RaftProtos.StateMachineLogEntryProto smLogEntry = RaftProtos.StateMachineLogEntryProto.newBuilder()
+ .setLogData(request.encode().getContent())
+ .build();
+ RaftProtos.LogEntryProto logEntry = RaftProtos.LogEntryProto.newBuilder()
+ .setTerm(1)
+ .setIndex(index)
+ .setStateMachineLogEntry(smLogEntry)
+ .build();
+ TransactionContext trx = mock(TransactionContext.class);
+ when(trx.getStateMachineLogEntry()).thenReturn(smLogEntry);
+ when(trx.getLogEntry()).thenReturn(logEntry);
+ return trx;
+ }
+
+ private static SCMStateMachine newStateMachine(SCMHADBTransactionBuffer buffer, ScmInvoker<?> invoker) {
+ StorageContainerManager scm = mock(StorageContainerManager.class);
+ when(scm.getMetrics()).thenReturn(SCMMetrics.create());
Review Comment:
Each call to this helper registers another global `SCMMetrics` source, but these new tests never unregister it. Since the helper is used by four test methods, later methods can fail with a duplicate metrics-source registration (the existing test explicitly unregisters its instance). Use a Mockito metrics instance here instead of registering a real source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org