You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "EdwinIngJ (via GitHub)" <gi...@apache.org> on 2025/10/09 02:38:08 UTC

[I] Nondeterministic Tests in KinesisSupervisorTest (druid)

EdwinIngJ opened a new issue, #18611:
URL: https://github.com/apache/druid/issues/18611

   I noticed some nondeterminism in the following tests:
   - `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplit` 
   - `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testMultiTask`. 
   
   Specifically, the problem in both tests comes from the tests assuming some ordering for the captured tasks.
   ### For `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplit`
   ```
   [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.696 s <<< FAILURE! -- in org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest
   [ERROR] org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplit -- Time elapsed: 0.692 s <<< FAILURE!
   java.lang.AssertionError: expected:<0> but was:<1>
           at org.junit.Assert.fail(Assert.java:89)
           at org.junit.Assert.failNotEquals(Assert.java:835)
           at org.junit.Assert.assertEquals(Assert.java:120)
           at org.junit.Assert.assertEquals(Assert.java:146)
           at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplitPhaseTwo(KinesisSupervisorTest.java:4476)
           at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplit(KinesisSupervisorTest.java:4182)
           at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
           at java.base/java.lang.reflect.Method.invoke(Method.java:580)
           at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
           at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
           at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
           at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
           at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
           at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
           at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
           at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
           at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
           at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
           at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
           at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
           at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
           at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
           at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
           at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
           at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
           at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
           at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
           at org.junit.runners.Suite.runChild(Suite.java:128)
           at org.junit.runners.Suite.runChild(Suite.java:27)
           at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
           at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
           at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
           at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
           at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
           at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
           at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
           at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:49)
           at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:120)
           at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:95)
           at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
           at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:69)
           at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:146)
           at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
           at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
           at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
           at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
   ```
   **Problem**: The `testShardSplit` test calls `testShardSplitPhaseTwo` and `testShardSplitPhaseThree`. Within these two function calls, there are assertions that check that the first element in the captured `postSplitTasks` list always has `TaskGroupId=0` and the second element always has `TaskGroupId=1`. However, the actual ordering depends on the ordering of `activelyReadingTaskGroups` ConcurrentHashMap in `SeekableStreamSupervisor`. More specifically, it depends on [this](https://github.com/apache/druid/blob/a4d0433e0bdf546cdcbbca1c54943ee6a5f7b3ef/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java#L3915) line in the `createNewTasks` where the `Task` gets created. Since the ordering of `.entrySet()` is not guaranteed, the order of the captured `postSplitTasks` list is also not guaranteed.
   
   I am able to consistently reproduce the test failure in `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testShardSplit`, by modifying line [3915](https://github.com/apache/druid/blob/a4d0433e0bdf546cdcbbca1c54943ee6a5f7b3ef/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java#L3915) in `createNewTasks` as follows:
   ```java
   -    for (Entry<Integer, TaskGroup> entry : activelyReadingTaskGroups.entrySet()) {
   +    List<Entry<Integer, TaskGroup>> sortedEntries = new ArrayList<>(activelyReadingTaskGroups.entrySet());
   +    sortedEntries.sort((e1, e2) -> Integer.compare(e2.getKey(), e1.getKey())); // sort entries in descending order
   +    for (Entry<Integer, TaskGroup> entry : sortedEntries) {
   ```
   
   Compiling and running `mvn -pl extensions-core/kinesis-indexing-service test -Dtest=org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest#testShardSplit` will result in the same error as the one shown above.
   
   
   ### For `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testMultiTask`
   ```
   [ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 2.717 s <<< FAILURE! -- in org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest
   [ERROR] org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testMultiTask -- Time elapsed: 0.982 s <<< FAILURE!
   java.lang.AssertionError: expected:<0> but was:<null>
           at org.junit.Assert.fail(Assert.java:89)
           at org.junit.Assert.failNotEquals(Assert.java:835)
           at org.junit.Assert.assertEquals(Assert.java:120)
           at org.junit.Assert.assertEquals(Assert.java:146)
           at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testMultiTask(KinesisSupervisorTest.java:609)
           at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
           at java.base/java.lang.reflect.Method.invoke(Method.java:580)
           at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
           at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
           at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
           at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
           at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
           at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
           at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
           at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
           at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
           at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
           at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
           at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
           at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
           at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
           at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
           at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
           at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
           at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
           at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
           at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
           at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
           at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
           at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
           at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
           at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
           at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
           at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
   ```
   
   **Problem**: Similar to the previous test, `testMultiTask` has assertions that assume some ordering for the captured `KinesisIndexTask`. Again, the actual ordering depends on the ordering of `activelyReadingTaskGroups` ConcurrentHashMap in `SeekableStreamSupervisor`. 
   
   I am also able to consistently reproduce the test failure in `org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testMultiTask`, using the same changes to line [3915](https://github.com/apache/druid/blob/a4d0433e0bdf546cdcbbca1c54943ee6a5f7b3ef/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java#L3915) mentioned above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Nondeterministic Tests in KinesisSupervisorTest (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz closed issue #18611: Nondeterministic Tests in KinesisSupervisorTest
URL: https://github.com/apache/druid/issues/18611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org