You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Nemesis123925 (via GitHub)" <gi...@apache.org> on 2023/11/25 01:33:05 UTC
[PR] fix by repalcing HashSet with TreeSet [pinot]
Nemesis123925 opened a new pull request, #12047:
URL: https://github.com/apache/pinot/pull/12047
## Setup:
Java version: openjdk 11.0.20.1
Maven version: Apache Maven 3.8.1
## Fixed Test:
org.apache.pinot.sql.parsers.rewriter.ExprMinMaxRewriterTest#testQueryRewrite
## Test Failure:
The test's `PARENT_EXPR_MIN` field's order shows nondeterminism when applying the `NonDex` tool. When asserting the created query's `PROJECT` field, the column names's order could be random, which `PARENT_EXPR_MIN(0,2,col1,col2,col5,col6)
` could randomly be `PARENT_EXPR_MIN(0,2,col1,col2,col6,col5)`.
## Root Cause:
The root cause is in [`appendParentExprMinMaxFunctions()` in ExprMinMaxRewriter.java](https://github.com/apache/pinot/blob/f79b618141e07c140663791959d96956ad91d811/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/ExprMinMaxRewriter.java#L99), which it uses `List.addAll()` function to add a `HashSet` `projectionColumns` into a `List`. This `List.addAll()` function will random iterate through the element in the `HashSet` and add them into the `List`, which cause the non-deternimism.
## Fix:
The Fix here is to first sort the fields by constructing a `TreeSet`, and use `List.addAll()` function to put the contruted `TreeSet` into the `List`. This can completely eleminite the non-deternimism inside of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] fix by repalcing HashSet with TreeSet [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12047:
URL: https://github.com/apache/pinot/pull/12047#issuecomment-1826188257
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12047?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`f79b618`)](https://app.codecov.io/gh/apache/pinot/commit/f79b618141e07c140663791959d96956ad91d811?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.62% compared to head [(`f4bad2d`)](https://app.codecov.io/gh/apache/pinot/pull/12047?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.74%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12047 +/- ##
=============================================
- Coverage 61.62% 46.74% -14.88%
+ Complexity 1151 945 -206
=============================================
Files 2386 1788 -598
Lines 129567 93954 -35613
Branches 20053 15192 -4861
=============================================
- Hits 79847 43921 -35926
- Misses 43913 46911 +2998
+ Partials 5807 3122 -2685
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-14.77%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-14.74%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-14.88%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-14.88%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.74% <100.00%> (-0.20%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12047/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12047?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] fix by repalcing HashSet with TreeSet [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12047:
URL: https://github.com/apache/pinot/pull/12047
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] fix by repalcing HashSet with TreeSet [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12047:
URL: https://github.com/apache/pinot/pull/12047#discussion_r1411204796
##########
pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/ExprMinMaxRewriter.java:
##########
@@ -107,7 +108,8 @@ private void appendParentExprMinMaxFunctions(boolean isMax, List<Expression> sel
operands.add(RequestUtils.getLiteralExpression((int) exprMinMaxFunctionIDMap.get(measuringColumns)));
operands.add(RequestUtils.getLiteralExpression(measuringColumns.size()));
operands.addAll(measuringColumns);
- operands.addAll(projectionColumns);
+ Set<Expression> sortedProjectionColumns = new TreeSet<>(projectionColumns);
Review Comment:
This is adding overhead to production code. Since the set is in general small, you may directly create `TreeSet` at first place (on line 169). To be more concise, you may change the code as following:
```
boolean added = exprMinMaxFunctionMap.computeIfAbsent(exprMinMaxMeasuringExpressions, k -> new TreeSet<>())
.add(exprMinMaxProjectionExpression);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org