You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "wuchong (via GitHub)" <gi...@apache.org> on 2025/08/24 08:16:05 UTC

[I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

wuchong opened a new issue, #1583:
URL: https://github.com/apache/fluss/issues/1583

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
   
   
   ### Fluss version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   https://github.com/apache/fluss/actions/runs/17186091094/job/48755270496?pr=1176
   
   ```
   Error:  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.408 s <<< FAILURE! - in com.alibaba.fluss.server.log.remote.CommitRemoteLogManifestITCase
   Error:  com.alibaba.fluss.server.log.remote.CommitRemoteLogManifestITCase.testDeleteOutOfSyncReplicaLogAfterCommit  Time elapsed: 1.004 s  <<< FAILURE!
   java.lang.AssertionError: 
   
   Expected size: 3 but was: 1 in:
   [LogSegment(baseOffset=0, size=0, lastModifiedTime=1756022776328, maxTimestamp=-1)]
   	at com.alibaba.fluss.server.log.remote.CommitRemoteLogManifestITCase.testDeleteOutOfSyncReplicaLogAfterCommit(CommitRemoteLogManifestITCase.java:99)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
   	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
   	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
   	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
   	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
   	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
   	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
   	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
   	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
   	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
   	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
   	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
   	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
   	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
   	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
   	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
   	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
   	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
   	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
   	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
   	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
   	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
   	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
   	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
   	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
   	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
   	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
   	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
   
   [INFO] Running com.alibaba.fluss.server.tablet.TabletServiceITCase
   [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.832 s - in com.alibaba.fluss.server.tablet.TabletServerFailOverITCase
   [INFO] Running com.alibaba.fluss.server.tablet.TabletServerITCase
   Warning:  Tests run: 13, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 12.48 s - in com.alibaba.fluss.server.tablet.TabletServiceITCase
   [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.386 s - in com.alibaba.fluss.server.tablet.TabletServerITCase
   [INFO] 
   [INFO] Results:
   [INFO] 
   Error:  Failures: 
   Error:    CommitRemoteLogManifestITCase.testDeleteOutOfSyncReplicaLogAfterCommit:99 
   Expected size: 3 but was: 1 in:
   [LogSegment(baseOffset=0, size=0, lastModifiedTime=1756022776328, maxTimestamp=-1)]
   [INFO] 
   Error:  Tests run: 56, Failures: 1, Errors: 0, Skipped: 2
   ```
   
   ### Solution
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3392727448

   > > [@luoyuxia](https://github.com/luoyuxia) thank you for the investigation! I think we can increase or remove the `log.replica.max-lag-time` configuration in the test, because we now supported controlled shutdown that the stopped tabletserver will send shutdown request to coordinator to trigger isr shrink at once without waiting the out-of-sync lag time. What do you think [@swuferhong](https://github.com/swuferhong) ?
   > 
   > If we want to remove `log.replica.max-lag-time`, the test cases also need to be modified. This is because the current test does not trigger a `tabletServer shutdown` but instead triggers a `replica stop` to mock replica out of sync. However, I feel that modifying the test to trigger a `tabletServer shutdown` would no longer be meaningful for this test.
   
   +1, if shut down tabletserver, the server won't receive the `NotifyRemoteLogOffsetsRequest` and then won't delete the segments that already copied to remote. But the test want to test although a follower replica is out of sync, if it leader commit remote log segment, the follower replica should still delete the log segment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3368868197

   @luoyuxia thank you for the investigation! I think we can increase or remove the `log.replica.max-lag-time` configuration in the test, because we now supported controlled shutdown that the stopped tabletserver will send shutdown request to coordinator to trigger isr shrink at once without waiting the out-of-sync lag time. What do you think @swuferhong ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3266326146

   After looking into the fail instance  https://github.com/apache/fluss/actions/runs/17547578108/job/49832509669?pr=1375
   Saw such logs:
   ```
   **16:36:55,281 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.LocalLog                         [] - Rolled new log segment at offset 0**
   **16:36:56,197 [fluss-scheduler-0-thread-2] INFO  org.apache.fluss.server.replica.Replica                      [] - Shrink ISR From [0, 1, 2] to [0]. Leader: (high watermark: 0, end offset: 10, out of sync replicas: [1, 2])**
   16:36:56,200 [fluss-netty-client(EPOLL)-61-3] INFO  org.apache.fluss.server.replica.Replica                      [] - ISR updated to [0] and bucket epoch updated to 1 for bucket TableBucket{tableId=0, bucket=0}
   **16:36:56,202 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.LocalLog                         [] - Rolled new log segment at offset 10**
   16:36:56,204 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.WriterStateManager               [] - Wrote writer snapshot at offset 10 with 0 producer ids for table bucket TableBucket{tableId=0, bucket=0} in 2 ms.
   **16:36:56,206 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.LocalLog                         [] - Rolled new log segment at offset 20**
   16:36:56,207 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.WriterStateManager               [] - Wrote writer snapshot at offset 20 with 0 producer ids for table bucket TableBucket{tableId=0, bucket=0} in 1 ms.
   16:36:56,208 [fluss-netty-server-worker-thread-3] INFO  org.apache.fluss.server.replica.fetcher.ReplicaFetcherManager [] - Remove fetcher for buckets: [TableBucket{tableId=0, bucket=0}]
   16:36:56,209 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.replica.fetcher.ReplicaFetcherManager [] - Remove fetcher for buckets: [TableBucket{tableId=0, bucket=0}]
   **16:36:56,210 [fluss-netty-server-worker-thread-1] INFO  org.apache.fluss.server.log.LocalLog                         [] - Rolled new log segment at offset 30**
   ```
   From the logs, we can know:
   - Firstly, it try to append 10 records according to log  `Rolled new log segment at offset 0`
   - The followers are out of sync according to log `Shrink ISR From [0, 1, 2] to [0]`
   - Then, it append the other records accoding to log `Rolled new log segment at offset 10`, `Rolled new log segment at offset 20*`
   
   Since the followers don't sync any records so that the verification to verify followers have 3 segments will fail. 
   Seems `log.replica.max-lag-time` is too short to wait the other followers to sync the log before  in this test case. Maybe we can increase it to 5 or 10 seconds. Although longer, should more stable.
   
   What's more, I check the log for success test case:
   ```
   **330 [ReplicaFetcherThread-0-2] INFO  org.apache.fluss.server.log.LocalLog [] - Rolled new log segment at offset 0**
   **7337 [fluss-netty-server-worker-thread-2] INFO  org.apache.fluss.server.log.LocalLog [] - Rolled new log segment at offset 10**
   ...
   **7870 [fluss-netty-server-worker-thread-2] INFO  org.apache.fluss.server.log.LocalLog [] - Rolled new log segment at offset 20**
   ...
   **11073 [fluss-scheduler-0-thread-8] INFO  org.apache.fluss.server.replica.Replica [] - Shrink ISR From [2, 1, 0] to [2]. Leader: (high watermark: 30, end offset: 40, out of sync replicas: [0, 1])**
   ```
   Shrink ISR(which is expected to be trigger by code ` FLUSS_CLUSTER_EXTENSION.stopReplica(stopFollower, tb, 1);`) should happen after append records successfully 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3368866370

   another instance: https://github.com/apache/fluss/actions/runs/18255910174/job/51977115685?pr=862


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-4167885961

   Another instance: https://github.com/apache/fluss/actions/runs/23831687003/job/69466447145


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3324836943

   @wuchong @luoyuxia can I pick this and try to resolve at my pace if it is not critical for the next release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3810215913

   Another instance: https://github.com/apache/fluss/actions/runs/21432199048/job/61714119853?pr=2331
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3383891529

   > [@luoyuxia](https://github.com/luoyuxia) thank you for the investigation! I think we can increase or remove the `log.replica.max-lag-time` configuration in the test, because we now supported controlled shutdown that the stopped tabletserver will send shutdown request to coordinator to trigger isr shrink at once without waiting the out-of-sync lag time. What do you think [@swuferhong](https://github.com/swuferhong) ?
   
   If we want to remove `log.replica.max-lag-time`, the test cases also need to be modified. This is because the current test does not trigger a `tabletServer shutdown` but instead triggers a `replica stop` to mock replica out of sync. However, I feel that modifying the test to trigger a `tabletServer shutdown` would no longer be meaningful for this test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3265712690

   another instance: https://github.com/apache/fluss/actions/runs/17547578108/job/49832509669?pr=1375


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3265705002

   another instance: https://github.com/apache/fluss/actions/runs/17544506465/job/49822733751?pr=1482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CommitRemoteLogManifestITCase#testDeleteOutOfSyncReplicaLogAfterCommit is unstable [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #1583:
URL: https://github.com/apache/fluss/issues/1583#issuecomment-3674068203

   Another instance: https://github.com/apache/fluss/actions/runs/20363795683/job/58514189381?pr=2063


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org