You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/09/17 15:18:11 UTC
[GitHub] [incubator-pinot] mr-agrwal opened a new issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
mr-agrwal opened a new issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028
Right now for some heavy Distinct Count Group By queries, we see a bottleneck in Broker performance.
This can be improved by converting the Broker Aggregation Phase from single-threaded to multi-threaded.
<img width="1237" alt="Screen Shot 2020-09-17 at 8 09 47 AM" src="https://user-images.githubusercontent.com/24796847/93490971-c3f66600-f926-11ea-9af7-dc43cd98a4d7.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv edited a comment on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
Posted by GitBox <gi...@apache.org>.
mayankshriv edited a comment on issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899
@mr-agrwal Could you please try the changes in this PR and provide some feedback: https://github.com/apache/incubator-pinot/pull/6044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv edited a comment on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
Posted by GitBox <gi...@apache.org>.
mayankshriv edited a comment on issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899
@mr-agrwal Could you please try the changes in this PR and provide some feedback: https://github.com/apache/incubator-pinot/pull/6044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899
@mr-agrwal Could you please try the changes in this PR and provide some feedback:
https://github.com/apache/incubator-pinot/pull/6044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-694310325
```
private void setGroupByResults(BrokerResponseNative brokerResponseNative, Collection<DataTable> dataTables) {
// Merge results from all data tables.
String[] columnNames = new String[_numAggregationFunctions];
Map<String, Object>[] intermediateResultMaps = new Map[_numAggregationFunctions];
if (_numGroupByExpressions == 1) {
for (DataTable dataTable : dataTables) {
for (int i = 0; i < _numAggregationFunctions; i++) {
if (columnNames[i] == null) {
columnNames[i] = dataTable.getString(i, 0);
intermediateResultMaps[i] = dataTable.getObject(i, 1);
} else {
mergeResultMap(intermediateResultMaps[i], dataTable.getObject(i, 1), _aggregationFunctions[i]);
}
}
}
} else {
for (DataTable dataTable : dataTables) {
for (int i = 0; i < _numAggregationFunctions; i++) {
if (columnNames[i] == null) {
columnNames[i] = dataTable.getString(i, 0);
intermediateResultMaps[i] = convertLegacyGroupKeyDelimiter(dataTable.getObject(i, 1));
} else {
mergeResultMap(intermediateResultMaps[i], convertLegacyGroupKeyDelimiter(dataTable.getObject(i, 1)),
_aggregationFunctions[i]);
}
}
}
}
```
Yes, it is single threaded. It's fine in most cases but we can definitely make it multi-threaded for the group by queries since the work done by the broker can be substantial
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #6028:
URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899
@mr-agrwal Could you please try the changes in this PR and provide some feedback:
https://github.com/apache/incubator-pinot/pull/6044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org