You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "yashmayya (via GitHub)" <gi...@apache.org> on 2025/01/20 04:16:53 UTC
[PR] Limit GroupByCombineOperator to use 2 * numCores threads instead of creating one task per operator [pinot]
yashmayya opened a new pull request, #14843:
URL: https://github.com/apache/pinot/pull/14843
- For most leaf stage combine operators (in SSQE), we calculate the number of tasks spawned as `Math.min(numSegments, maxExecutionThreads)` where `maxExecutionThreads` is calculated as `Math.max(1, Math.min(10, numCores / 2))` (unless it is explicitly specified in the query options using `maxExecutionThreads`).
- Each of these tasks processes a subset of the segments. However, for the `GroupByCombineOperator` specifically, this `maxExecutionThreads` is currently overridden to be equal to the number of segments being processed (see [here](https://github.com/apache/pinot/blob/d1ac83e98020810560acb0c88a6fb7475c596252/pinot-core/src/main/java/org/apache/pinot/core/operator/combine/GroupByCombineOperator.java#L82-L91) by default for some reason.
- This is not so problematic for the single-stage query engine where these tasks are submitted to a fixed thread pool executor service with the number of threads being `2 * numCores` by default ([this one](https://github.com/apache/pinot/blob/d1ac83e98020810560acb0c88a6fb7475c596252/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/resources/ResourceManager.java#L94-L95)).
- In fact, some of the combine operators even have comments stating that while the `numTasks` are the number of async tasks submitted to the executor service, the actual number of threads used by the server would be limited to the fixed number of threads available in the executor service.
- However, in the multi-stage engine's leaf stages, the executor service used here is the same one from `QueryRunner` - i.e., [this cached thread pool executor](https://github.com/apache/pinot/blob/fbbabd43f2d8ef6f8e4966d8f7f60dc88dfcede8/pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java#L167-L169) that can spawn infinite threads. This means that for leaf stage operators that involve a group by, the number of threads spawned for a single multi-stage query on each server can be equal to the number of segments being processed on that server (could be thousands of threads which is super problematic).
- This patch updates the override to cap the number of tasks spawned by the `GroupByCombineOperator` by default to be the default number of query worker threads (which is `2 * numCores`). This shouldn't impact SSQE performance while also avoiding the creation of a huge number of threads in MSQE.
- Note that this is only a bandaid solution and we need to more carefully think through all the implications of the threading model for the MSQE (cc @gortiz who has some interesting ideas here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit GroupByCombineOperator to use 2 * numCores threads instead of creating one task per operator [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on PR #14843:
URL: https://github.com/apache/pinot/pull/14843#issuecomment-2605174430
> Can we verify that empirically with a test?
I updated the existing `BenchmarkQueries` to also support a larger number of segments and here's the results for the group by queries in that benchmark.
Old (without changes from this PR):
```
Benchmark (_numRows) (_numSegments) (_query) (_scenario) Mode Cnt Score Error Units
BenchmarkQueries.query 1500000 1 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 36.764 ± 1.189 ms/op
BenchmarkQueries.query 1500000 1 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 134.669 ± 2.014 ms/op
BenchmarkQueries.query 1500000 1 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 5.147 ± 0.334 ms/op
BenchmarkQueries.query 1500000 1 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 3.568 ± 0.084 ms/op
BenchmarkQueries.query 1500000 1 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 35.912 ± 1.409 ms/op
BenchmarkQueries.query 1500000 2 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 39.488 ± 0.565 ms/op
BenchmarkQueries.query 1500000 2 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 157.133 ± 3.754 ms/op
BenchmarkQueries.query 1500000 2 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 5.430 ± 0.158 ms/op
BenchmarkQueries.query 1500000 2 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 3.731 ± 0.163 ms/op
BenchmarkQueries.query 1500000 2 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 36.784 ± 1.647 ms/op
BenchmarkQueries.query 1500000 10 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 166.385 ± 10.191 ms/op
BenchmarkQueries.query 1500000 10 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 735.311 ± 20.755 ms/op
BenchmarkQueries.query 1500000 10 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 25.390 ± 0.409 ms/op
BenchmarkQueries.query 1500000 10 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 16.623 ± 0.327 ms/op
BenchmarkQueries.query 1500000 10 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 175.212 ± 3.031 ms/op
BenchmarkQueries.query 1500000 50 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 961.633 ± 17.295 ms/op
BenchmarkQueries.query 1500000 50 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 3640.062 ± 21.391 ms/op
BenchmarkQueries.query 1500000 50 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 123.276 ± 1.000 ms/op
BenchmarkQueries.query 1500000 50 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 81.729 ± 0.784 ms/op
BenchmarkQueries.query 1500000 50 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 862.859 ± 49.178 ms/op
```
New (with changes from this PR):
```
Benchmark (_numRows) (_numSegments) (_query) (_scenario) Mode Cnt Score Error Units
BenchmarkQueries.query 1500000 1 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 37.356 ± 0.371 ms/op
BenchmarkQueries.query 1500000 1 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 133.085 ± 1.483 ms/op
BenchmarkQueries.query 1500000 1 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 5.282 ± 0.309 ms/op
BenchmarkQueries.query 1500000 1 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 3.648 ± 0.147 ms/op
BenchmarkQueries.query 1500000 1 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 34.617 ± 0.604 ms/op
BenchmarkQueries.query 1500000 2 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 37.743 ± 0.558 ms/op
BenchmarkQueries.query 1500000 2 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 157.974 ± 1.498 ms/op
BenchmarkQueries.query 1500000 2 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 5.401 ± 0.134 ms/op
BenchmarkQueries.query 1500000 2 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 3.633 ± 0.145 ms/op
BenchmarkQueries.query 1500000 2 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 34.755 ± 1.062 ms/op
BenchmarkQueries.query 1500000 10 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 191.722 ± 4.703 ms/op
BenchmarkQueries.query 1500000 10 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 762.582 ± 5.516 ms/op
BenchmarkQueries.query 1500000 10 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 24.967 ± 0.320 ms/op
BenchmarkQueries.query 1500000 10 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 16.709 ± 0.314 ms/op
BenchmarkQueries.query 1500000 10 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 175.788 ± 11.362 ms/op
BenchmarkQueries.query 1500000 50 SELECT RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_INT_COL,INT_COL EXP(0.5) avgt 5 917.493 ± 17.065 ms/op
BenchmarkQueries.query 1500000 50 SELECT RAW_STRING_COL,RAW_INT_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY RAW_STRING_COL,RAW_INT_COL,INT_COL EXP(0.5) avgt 5 3623.057 ± 87.547 ms/op
BenchmarkQueries.query 1500000 50 SELECT NO_INDEX_STRING_COL,INT_COL,COUNT(*) FROM MyTable GROUP BY NO_INDEX_STRING_COL,INT_COL ORDER BY NO_INDEX_STRING_COL, INT_COL ASC EXP(0.5) avgt 5 122.368 ± 1.879 ms/op
BenchmarkQueries.query 1500000 50 SELECT NO_INDEX_STRING_COL,LOW_CARDINALITY_STRING_COL,COUNT(*) FROM MyTable GROUP BY LOW_CARDINALITY_STRING_COL,NO_INDEX_STRING_COL ORDER BY LOW_CARDINALITY_STRING_COL, NO_INDEX_STRING_COL ASC EXP(0.5) avgt 5 82.259 ± 2.439 ms/op
BenchmarkQueries.query 1500000 50 select count(*), year(INT_COL) as y, month(INT_COL) as m from MyTable group by y, m EXP(0.5) avgt 5 781.819 ± 13.464 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit GroupByCombineOperator to use 2 * numCores threads instead of creating one task per operator [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14843:
URL: https://github.com/apache/pinot/pull/14843#issuecomment-2601365771
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14843?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`e440a64`)](https://app.codecov.io/gh/apache/pinot/commit/e440a64a10719e9e55128eafb75fc25161ffaaef?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1596 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (e440a64). Click for more details.
>
> <details><summary>HEAD has 53 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (e440a64) |
>|------|------|------|
>|integration|7|1|
>|integration2|3|1|
>|temurin|12|1|
>|java-21|7|1|
>|skip-bytebuffers-true|3|1|
>|skip-bytebuffers-false|7|0|
>|unittests|5|0|
>|unittests1|2|0|
>|java-11|5|0|
>|unittests2|3|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #14843 +/- ##
=============================================
- Coverage 61.75% 0.00% -61.76%
=============================================
Files 2436 3 -2433
Lines 133233 6 -133227
Branches 20636 0 -20636
=============================================
- Hits 82274 0 -82274
+ Misses 44911 6 -44905
+ Partials 6048 0 -6048
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/14843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/14843?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit GroupByCombineOperator to use 2 * numCores threads instead of creating one task per operator [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #14843:
URL: https://github.com/apache/pinot/pull/14843#issuecomment-2601887090
> This shouldn't impact SSQE performance while ...
Can we verify that empirically with a test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit GroupByCombineOperator to use 2 * numCores threads instead of creating one task per operator [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #14843:
URL: https://github.com/apache/pinot/pull/14843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org