You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Nemesis123925 (via GitHub)" <gi...@apache.org> on 2023/11/02 18:36:22 UTC
[I] [Flaky-Test] Fix Flaky tests in Pinot-Core org.apache.pinot.queries.ExplainPlanQueriesTest [pinot]
Nemesis123925 opened a new issue, #11939:
URL: https://github.com/apache/pinot/issues/11939
## Setup:
Java version: openjdk 11.0.20.1
Maven version: Apache Maven 3.8.1
## Fixed Test:
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelect
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregate
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterGroupBy
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterGroupByVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndex
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupBy
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupByHaving
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupByHavingVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupByOrderBy
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupByOrderByVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexGroupByVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterIndexVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterOnTextIndexColumn
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateUsingFilterOnTextIndexColumnVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectAggregateVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnUsingFilterOnJsonIndexColumnVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnUsingFilterOnRangeIndexColumn
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnUsingFilterOnRangeIndexColumnVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsUsingFilter
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsUsingFilterOnInvertedIndexColumn
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsUsingFilterOnInvertedIndexColumnVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsUsingFilterVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsVariationsOfAndOperators
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsVariationsOfAndOperatorsVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsVariationsOfOrOperators
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectColumnsVariationsOfOrOperatorsVerbose
org.apache.pinot.queries.ExplainPlanQueriesTest#testSelectVerbose
## Test Failure:
The test's `PROJECT` field's order shows nondeterminism when applying the `NonDex` tool. When asserting the created query's `PROJECT` field, the column names's order could be random, which `PROJECT(invertedIndexCol1, noIndexCol1)
` could randomly be `PROJECT(noIndexCol1, invertedIndexCol1)`. Also, when the create query consisted of nested queries, the two nested queries's order could also be random.
## Root Cause:
The root cause is in [pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java's `toExplainString()` function](https://github.com/Nemesis123925/pinot/blob/24af80d3bccdb4c2b86cae392ea022b824db1c1a/pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java#L86), which it iterate through the `keySet` of `_dataSourceMap`, which is created as a `HashMap`. This `_dataSourceMap` is created in [pinot-core/src/main/java/org/apache/pinot/core/plan/ProjectPlanNode.java](https://github.com/Nemesis123925/pinot/blob/ad0d2b173a67ac19bc39120179f48192c80520e4/pinot-core/src/main/java/org/apache/pinot/core/plan/ProjectPlanNode.java#L74) and its order is determined by [pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.java](https://github.com/Nemesis123925/pinot/blob/ad0d2b173a67ac19bc39120179f48192c80520e4/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.jav
a#L88), which are whether `HashMap` or `HashSet`
## Fix:
The Fix here is by replacing the `HashSet` by `LinkedHashSet` and `HashMap` by `LinkedHashSet` in the above position. This complete eliminates the nondeterminism inside. This will also disturb the hardcoded `result` in the tests, so I also change the order in the `result` array in the tests, and make the tests pass. However, I am here to ask is it acceptable to change the main code? If not, I can replace the fix by only changing the test code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Flaky-Test] Fix Flaky tests in Pinot-Core org.apache.pinot.queries.ExplainPlanQueriesTest [pinot]
Posted by "Nemesis123925 (via GitHub)" <gi...@apache.org>.
Nemesis123925 commented on issue #11939:
URL: https://github.com/apache/pinot/issues/11939#issuecomment-1791694331
Yes, Here is the [pull request](https://github.com/apache/pinot/pull/11941)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Flaky-Test] Fix Flaky tests in Pinot-Core org.apache.pinot.queries.ExplainPlanQueriesTest [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11939:
URL: https://github.com/apache/pinot/issues/11939#issuecomment-1791690081
Thanks for raising the issue with the necessary fix. Can you open a pull request with the code change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org