You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Shiyang-Zhao (via GitHub)" <gi...@apache.org> on 2025/10/10 03:54:51 UTC
[PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Shiyang-Zhao opened a new pull request, #18621:
URL: https://github.com/apache/druid/pull/18621
This PR fixes nondeterministic behavior in the following flaky tests:
- org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent
- org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testIsTaskCurrent
### Description
These tests occasionally failed due to nondeterministic behavior in how the supervisor evaluated whether a task was “current.”
The issue originated from unstable iteration order inside the `DataSchema` serialization process, which affected equality checks between supervisor and task configurations.
```
KafkaSupervisorTest.testIsTaskCurrent()
↓
KafkaSupervisor.isTaskCurrent()
↓
SeekableStreamSupervisor.isTaskCurrent()
↓
SeekableStreamSupervisor.generateSequenceName()
↓
ObjectMapper.writeValueAsString(DataSchema)
↓
DataSchema → DimensionsSpec.dimensionExclusions used a HashSet (non-deterministic iteration)
```
Here is a sample of an error from running **org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent**:
```
[ERROR] org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent[numThreads = 1] -- Time elapsed: 0.79 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent(KafkaSupervisorTest.java:4740)
[ERROR] org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent[numThreads = 8] -- Time elapsed: 0.04 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorTest.testIsTaskCurrent(KafkaSupervisorTest.java:4741)
```
**Problem:**
`SeekableStreamSupervisor.isTaskCurrent()` verifies task consistency by recomputing a sequence name hash using `generateSequenceName()`, which serializes the `DataSchema` object to JSON. Within `DataSchema`, the `DimensionsSpec` class maintained its `dimensionExclusions` field as a `HashSet`, which does not guarantee element iteration order. Because JSON serialization depends on iteration order, identical `DimensionsSpec` instances could produce different serialized strings across runs. This caused the computed hashes for otherwise equivalent `DataSchema` objects to differ, leading `isTaskCurrent()` to incorrectly treat valid tasks as outdated or inconsistent. The issue manifested as nondeterministic test outcomes, since the JSON field order—and thus the resulting hash—varied between executions.
**Proposed Changes:**
- Replaced `HashSet` with `LinkedHashSet` in `DimensionsSpec` constructor to ensure deterministic iteration order.
- Ensured `DataSchema` serialization and equality checks are now stable and repeatable.
Here is a sample of an error from running **org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testIsTaskCurrent**:
```
[ERROR] org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testIsTaskCurrent[numThreads = 1] -- Time elapsed: 0.84 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisorTest.testIsTaskCurrent(KinesisSupervisorTest.java:4821)
```
**Problem:**
The same serialization nondeterminism affected Kinesis supervisor tests, since `KinesisSupervisor` extends the same
`SeekableStreamSupervisor` base class and reuses the same `isTaskCurrent()` logic.
**Proposed Changes:**
- No Kinesis-specific changes were required; the fix in `DimensionsSpec` resolved both cases.
---
This PR has:
- [x] been self-reviewed.
- [x] ensured no production logic changes beyond test stabilization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "Shiyang-Zhao (via GitHub)" <gi...@apache.org>.
Shiyang-Zhao commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3453717728
Hi @kfaraz,
I’ve reverted the previous code changes and only added a brief comment explaining the behavior, as suggested.
Please let me know if you’d like me to adjust the wording or phrasing of the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "Shiyang-Zhao (via GitHub)" <gi...@apache.org>.
Shiyang-Zhao commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3409775142
Hi @kfaraz,
I’ve updated the code following your feedback — replaced the `LinkedHashSet` with a `List` as suggested.
I intentionally kept the existing function signature and return type unchanged because they’re used in multiple other files, and changing them would require broader refactoring.
Please let me know if this matches what you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3449351918
Thanks for sharing the details, @Shiyang-Zhao !
> I could look into mitigating this directly within the test code, but given that the sequence hash is computed within the code under test, there may be limited options. Would you still like me to proceed?
Do you mean we would be able to handle the discrepancies only within the test and not in the production code?
If yes, then I don't think that would add a lot of value.
Ideally, we would want a test to prove that running production supervisors will not be affected.
If the change in this PR is certain or even likely to affect running supervisors when the Overlord restarts, we would need to rethink the solution.
But as you mention, since the order remains the same when running in ONE mode, I think we can live with the status quo for now. We could update this PR to just leave a comment in the relevant code.
Hope that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3826743452
This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "Shiyang-Zhao (via GitHub)" <gi...@apache.org>.
Shiyang-Zhao commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3440063873
Hi @kfaraz ,
Apologies for the delayed response. I was able to identify that the test is indeed flaky when using Nondex (https://github.com/TestingResearchIllinois/NonDex), which randomizes the iteration order of unordered collections such as Set. Specifically, the testIsTaskCurrent() test failed under FULL mode, where the iteration order is randomized on every collection traversal. However, when running in ONE mode—where the iteration order is randomized only once per JVM instance—the test passed consistently.
This behavior occurs because the generateSequenceName method depends on the ordering of the serialized JSON representation of the dimensionExclusions set. In ONE mode, the ordering remains stable across calls, resulting in consistent sequence names. While this makes it difficult to reproduce the failure simply by adding elements to the set, it’s worth noting that since the Set interface provides no ordering guarantees, this could still occur under certain conditions or in future JDK versions.
Regarding your concern about upgrading the Overlord and potentially affecting currently running supervisors, I could look into mitigating this directly within the test code, but given that the sequence hash is computed within the code under test, there may be limited options. Would you still like me to proceed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3405152603
Yes, @Shiyang-Zhao , since we are touching this code, we might as well change it to a List since the constructor already accepts a list. We should remove the discrepancy between the serialization and the deserialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3392998861
Thanks for catching this, @Shiyang-Zhao !
This must have been causing CI failures repeatedly.
Have you come across any failed GitHub action due to this bug? Could you please share the link for the same?
Also, instead of a LinkedHashSet, I think we should just use a List since the field `dimensionExclusions` is already deserialized as a `List<String>` in the `@JsonCreator` constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18621: Ensure ordering in dimension exclusion using LinkedHashSet
URL: https://github.com/apache/druid/pull/18621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3706527340
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "Shiyang-Zhao (via GitHub)" <gi...@apache.org>.
Shiyang-Zhao commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3399646065
Hi @kfaraz
I wasn’t able to find a CI build that currently fails due to this issue, but the ordering of a `HashSet` (which is a `@JsonProperty` in this case) is not guaranteed. This can potentially lead to nondeterministic behavior in the future, since `SeekableStreamSupervisor.generateSequenceName()` computes a hash code based on the serialized JSON.
A `LinkedHashSet` was used since modifying the field `dimensionExclusions` would require changes to other parts of the code that rely on set operations. For example, `dimensionExclusions` is used in `verify` on line 293 in `DimensionsSpec.java` to perform a set intersection.
Would you still like me to change it to a `List`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Ensure ordering in dimension exclusion using LinkedHashSet (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18621:
URL: https://github.com/apache/druid/pull/18621#issuecomment-3417933382
> I wasn’t able to find a CI build that currently fails due to this issue, but the ordering of a HashSet (which is a @JsonProperty in this case) is not guaranteed. This can potentially lead to nondeterministic behavior in the future, since SeekableStreamSupervisor.generateSequenceName() computes a hash code based on the serialized JSON.
@Shiyang-Zhao , I looked at this some more.
I also tried to run the `KafkaSupervisorTest.testIsTaskCurrent()` until failure but it didn't flake out.
Were you able to cause it to fail by adding some items to `dimensionExclusions`?
If yes, could you please add the failing test in this PR?
A concern with changing the `dimensionExclusions` to either a sorted set or even a list is that when the Overlord is upgraded, we could adversely affect currently running supervisors. We would cause a different sequence name to be generated for the same supervisor thus causing the same problems (i.e. segment allocation and determining if a task is current) that we are trying to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org