You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/08/31 08:04:07 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11472: Stream leaf stage blocks
Jackie-Jiang opened a new pull request, #11472:
URL: https://github.com/apache/pinot/pull/11472
- Make multi-stage leaf operator use the streaming operator from single-stage engine to avoid memory issue and also improve efficiency.
- Applied back pressure to the streaming operator
- Fix the execution stats override issue
# Documentation
Added new query option `maxLeafPendingBlocks` to control the pending blocks (back pressure, 100 by default)
# Incompatible
The following interface changed:
- BaseResultsBlock
- PlanMaker
- QueryExecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11472: Stream leaf stage blocks
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #11472:
URL: https://github.com/apache/pinot/pull/11472#discussion_r1313204421
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/AggregationResultsBlock.java:
##########
@@ -45,10 +45,13 @@
public class AggregationResultsBlock extends BaseResultsBlock {
private final AggregationFunction[] _aggregationFunctions;
private final List<Object> _results;
+ private final QueryContext _queryContext;
- public AggregationResultsBlock(AggregationFunction[] aggregationFunctions, List<Object> results) {
+ public AggregationResultsBlock(AggregationFunction[] aggregationFunctions, List<Object> results,
+ QueryContext queryContext) {
_aggregationFunctions = aggregationFunctions;
_results = results;
+ _queryContext = queryContext;
Review Comment:
Does it make sense to put queryContext into BaseResultsBlock with a constructor and use super here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11472: Stream leaf stage blocks
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11472:
URL: https://github.com/apache/pinot/pull/11472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11472: Stream leaf stage blocks
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11472:
URL: https://github.com/apache/pinot/pull/11472#discussion_r1313332875
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/AggregationResultsBlock.java:
##########
@@ -45,10 +45,13 @@
public class AggregationResultsBlock extends BaseResultsBlock {
private final AggregationFunction[] _aggregationFunctions;
private final List<Object> _results;
+ private final QueryContext _queryContext;
- public AggregationResultsBlock(AggregationFunction[] aggregationFunctions, List<Object> results) {
+ public AggregationResultsBlock(AggregationFunction[] aggregationFunctions, List<Object> results,
+ QueryContext queryContext) {
_aggregationFunctions = aggregationFunctions;
_results = results;
+ _queryContext = queryContext;
Review Comment:
I considered that, but it doesn't apply to metadata and exception block, so I kept it only in data blocks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11472: Stream leaf stage blocks
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11472:
URL: https://github.com/apache/pinot/pull/11472#issuecomment-1700624651
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11472](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b6ad16e) into [master](https://app.codecov.io/gh/apache/pinot/commit/b3b0b9ea8ab99e66cb77afdd8eabc1bb03cb448b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b3b0b9e) will **decrease** coverage by `48.50%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11472 +/- ##
=============================================
- Coverage 62.92% 14.43% -48.50%
+ Complexity 1108 201 -907
=============================================
Files 2318 2319 +1
Lines 124321 124380 +59
Branches 18980 18988 +8
=============================================
- Hits 78234 17952 -60282
- Misses 40534 104911 +64377
+ Partials 5553 1517 -4036
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.43% <0.00%> (-35.54%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.43% <0.00%> (-48.50%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.43% <0.00%> (-48.50%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11472/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.43% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../apache/pinot/common/exception/QueryException.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-93.55%)` | :arrow_down: |
| [...e/pinot/common/utils/config/QueryOptionsUtils.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvY29uZmlnL1F1ZXJ5T3B0aW9uc1V0aWxzLmphdmE=) | `0.00% <0.00%> (-69.36%)` | :arrow_down: |
| [.../pinot/core/operator/InstanceResponseOperator.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9JbnN0YW5jZVJlc3BvbnNlT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-88.38%)` | :arrow_down: |
| [...ot/core/operator/blocks/InstanceResponseBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvSW5zdGFuY2VSZXNwb25zZUJsb2NrLmphdmE=) | `0.00% <0.00%> (-75.52%)` | :arrow_down: |
| [...erator/blocks/results/AggregationResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9BZ2dyZWdhdGlvblJlc3VsdHNCbG9jay5qYXZh) | `0.00% <0.00%> (-61.80%)` | :arrow_down: |
| [...core/operator/blocks/results/BaseResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9CYXNlUmVzdWx0c0Jsb2NrLmphdmE=) | `0.00% <ø> (-93.88%)` | :arrow_down: |
| [.../operator/blocks/results/DistinctResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9EaXN0aW5jdFJlc3VsdHNCbG9jay5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...operator/blocks/results/ExceptionResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9FeGNlcHRpb25SZXN1bHRzQmxvY2suamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
| [...e/operator/blocks/results/ExplainResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9FeHBsYWluUmVzdWx0c0Jsb2NrLmphdmE=) | `0.00% <0.00%> (-70.38%)` | :arrow_down: |
| [...e/operator/blocks/results/GroupByResultsBlock.java](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvcmVzdWx0cy9Hcm91cEJ5UmVzdWx0c0Jsb2NrLmphdmE=) | `0.00% <0.00%> (-83.20%)` | :arrow_down: |
| ... and [35 more](https://app.codecov.io/gh/apache/pinot/pull/11472?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [1447 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11472/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org