You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "vrajat (via GitHub)" <gi...@apache.org> on 2025/06/23 07:40:24 UTC
[PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
vrajat opened a new pull request, #16172:
URL: https://github.com/apache/pinot/pull/16172
The following logging improvements have been made to the OOM Protection module:
* Convert thread registration & de-registration logs to DEBUG
* Add a log function to take snapshot of the state when queries are killed. Previously these logs were inline. Also the snapshot was not logged in `PerQueryCPUMemAccountant.WatcherTask.killAllQueries`.
An unrelated improvement is that in the `else` block of `PerQueryCPUMemAccountant.WatcherTask.killMostExpensiveQuery`, cpu time was considered if memory sampling is off. This can lead to serious jeopardy. The block is executed if `accounting.oom.enable.killing.query` is true AND `accounting.enable.thread.memory.sampling`. Rather this invalid configuration should be ignored or handled correctly. So the if block has been removed.
Closes #16111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16172:
URL: https://github.com/apache/pinot/pull/16172#discussion_r2162616542
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
Review Comment:
We now emit two WARNs per kill: one line for each terminated query plus a second line with the whole map. Does making the “aggregation results” line INFO (or even DEBUG) help so we can keep WARN dedicated to actionable events.
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
Review Comment:
Looks like `killAllQueries()` loops over every entry in queryResources and logs each individually. For clusters in meltdown that could still flood logs. Do we want to cap this at some N entries?
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
+ }
+
+ protected void logTerminatedQuery(QueryResourceTracker queryResourceTracker, long totalHeapMemoryUsage) {
Review Comment:
nit : This helper are declared protected but only used inside the same inner class; would private would convey intent better (and allow inlining in future clean-ups).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16172:
URL: https://github.com/apache/pinot/pull/16172#issuecomment-2995570842
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16172?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `18 lines` in your changes missing coverage. Please review.
> Project coverage is 33.40%. Comparing base [(`1a476de`)](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`7b6d9c1`)](https://app.codecov.io/gh/apache/pinot/commit/7b6d9c1cb1d49454d01a311812644b2112f935b2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 315 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16172?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...re/accounting/PerQueryCPUMemAccountantFactory.java](https://app.codecov.io/gh/apache/pinot/pull/16172?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Faccounting%2FPerQueryCPUMemAccountantFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9hY2NvdW50aW5nL1BlclF1ZXJ5Q1BVTWVtQWNjb3VudGFudEZhY3RvcnkuamF2YQ==) | 0.00% | [18 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16172?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (1a476de) and HEAD (7b6d9c1). Click for more details.
>
> <details><summary>HEAD has 68 uploads less than BASE</summary>
>
>| Flag | BASE (1a476de) | HEAD (7b6d9c1) |
>|------|------|------|
>|java-21|10|1|
>|unittests|6|1|
>|unittests1|3|0|
>|skip-bytebuffers-true|4|0|
>|temurin|15|1|
>|unittests2|3|1|
>|skip-bytebuffers-false|8|0|
>|java-11|5|0|
>|integration|9|0|
>|integration2|3|0|
>|integration1|3|0|
>|custom-integration1|3|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #16172 +/- ##
=============================================
- Coverage 62.90% 33.40% -29.50%
+ Complexity 1386 659 -727
=============================================
Files 2867 2953 +86
Lines 163354 169953 +6599
Branches 24952 25999 +1047
=============================================
- Hits 102755 56770 -45985
- Misses 52847 108624 +55777
+ Partials 7752 4559 -3193
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.40% <0.00%> (-29.42%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.40% <0.00%> (-29.50%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.40% <0.00%> (-29.50%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16172/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.40% <0.00%> (-0.17%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16172?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16172:
URL: https://github.com/apache/pinot/pull/16172#discussion_r2162611002
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -892,35 +908,16 @@ private void killMostExpensiveQuery() {
" Query %s got killed because using %d bytes of memory on %s: %s, exceeding the quota",
maxUsageTuple._queryId, maxUsageTuple.getAllocatedBytes(), _instanceType, _instanceId)));
interruptRunnerThread(maxUsageTuple.getAnchorThread());
- LOGGER.error("Query {} got picked because using {} bytes of memory, actual kill committed true}",
- maxUsageTuple._queryId, maxUsageTuple._allocatedBytes);
- LOGGER.error("Current task status recorded is {}", _threadEntriesMap);
+ logTerminatedQuery(maxUsageTuple, _usedBytes);
} else if (!_oomKillQueryEnabled) {
LOGGER.warn("Query {} got picked because using {} bytes of memory, actual kill committed false "
+ "because oomKillQueryEnabled is false",
maxUsageTuple._queryId, maxUsageTuple._allocatedBytes);
} else {
LOGGER.warn("But all queries are below quota, no query killed");
}
- } else {
- maxUsageTuple = Collections.max(_aggregatedUsagePerActiveQuery.values(),
- Comparator.comparing(AggregatedStats::getCpuTimeNs));
- if (_oomKillQueryEnabled) {
Review Comment:
Looks like this was dead-code and never reached, good catch to remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16172:
URL: https://github.com/apache/pinot/pull/16172#discussion_r2162601621
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
Review Comment:
nit : `toString` can be removed
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -892,35 +908,16 @@ private void killMostExpensiveQuery() {
" Query %s got killed because using %d bytes of memory on %s: %s, exceeding the quota",
maxUsageTuple._queryId, maxUsageTuple.getAllocatedBytes(), _instanceType, _instanceId)));
interruptRunnerThread(maxUsageTuple.getAnchorThread());
- LOGGER.error("Query {} got picked because using {} bytes of memory, actual kill committed true}",
- maxUsageTuple._queryId, maxUsageTuple._allocatedBytes);
- LOGGER.error("Current task status recorded is {}", _threadEntriesMap);
+ logTerminatedQuery(maxUsageTuple, _usedBytes);
} else if (!_oomKillQueryEnabled) {
LOGGER.warn("Query {} got picked because using {} bytes of memory, actual kill committed false "
+ "because oomKillQueryEnabled is false",
maxUsageTuple._queryId, maxUsageTuple._allocatedBytes);
} else {
LOGGER.warn("But all queries are below quota, no query killed");
}
- } else {
- maxUsageTuple = Collections.max(_aggregatedUsagePerActiveQuery.values(),
- Comparator.comparing(AggregatedStats::getCpuTimeNs));
- if (_oomKillQueryEnabled) {
Review Comment:
Looks like this was dead-code and never reached, nice to remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #16172:
URL: https://github.com/apache/pinot/pull/16172#discussion_r2161223045
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
+ }
+
+ protected void logTerminatedQuery(QueryResourceTracker queryResourceTracker, long totalHeapMemoryUsage) {
+ LOGGER.error("Query {} terminated. Memory Usage: {}. Cpu Usage: {}. Total Heap Usage: {}",
Review Comment:
[nitpick] Consider whether an ERROR log level is appropriate for logging query terminations. If termination is an expected outcome due to OOM conditions, switching to a WARN level might better reflect the operational scenario.
```suggestion
LOGGER.warn("Query {} terminated. Memory Usage: {}. Cpu Usage: {}. Total Heap Usage: {}",
```
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -850,6 +861,11 @@ void killAllQueries() {
System.gc();
_numQueriesKilledConsecutively = 0;
}
+ Map<String, ? extends QueryResourceTracker> queryResources = getQueryResources();
Review Comment:
Consider reusing the 'queryResources' variable for the subsequent logQueryResourceUsage call to avoid multiple evaluations of getQueryResources() and ensure consistent logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #16172:
URL: https://github.com/apache/pinot/pull/16172#discussion_r2163757738
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
Review Comment:
This is the same as before.
`_threadEntriesMap` is not useful in my experience. Query Aggregation results have been useful to know the other queries running in the server. I'll remove the log line for `_threadEntriesMap`. Tell me if you want it back.
I consolidate the following log lines into this function.
https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java#L897
https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java#L923
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
Review Comment:
Good point. I'll not log queries at all. This will be similar to the current state.
##########
pinot-core/src/main/java/org/apache/pinot/core/accounting/PerQueryCPUMemAccountantFactory.java:
##########
@@ -482,6 +482,17 @@ public Map<String, AggregatedStats> aggregate(boolean isTriggered) {
public void postAggregation(Map<String, AggregatedStats> aggregatedUsagePerActiveQuery) {
}
+ protected void logQueryResourceUsage(Map<String, ? extends QueryResourceTracker> aggregatedUsagePerActiveQuery) {
+ LOGGER.warn("Current task status recorded is {}", _threadEntriesMap);
+ LOGGER.warn("Query aggregation results {} for the previous kill.", aggregatedUsagePerActiveQuery.toString());
+ }
+
+ protected void logTerminatedQuery(QueryResourceTracker queryResourceTracker, long totalHeapMemoryUsage) {
Review Comment:
We dont want these logs. We have a Postgres system table inspired logging framework. So we want to log this info into that system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Reduce logs and improve logging when queries are terminated due to OOM. [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #16172:
URL: https://github.com/apache/pinot/pull/16172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org