You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/08/14 22:53:42 UTC
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Jackie-Jiang opened a new pull request #5867:
URL: https://github.com/apache/incubator-pinot/pull/5867
## Description
Add `PostAggregationHandler` to handle the post-aggregation calculation
Enhance `AggregationDataTableReducer` and `GroupByDataTableReducer` to support post-aggregation in SELECT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Posted by GitBox <gi...@apache.org>.
npawar commented on a change in pull request #5867:
URL: https://github.com/apache/incubator-pinot/pull/5867#discussion_r472381824
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
##########
@@ -160,103 +160,49 @@ public void reduceAndSetResults(String tableName, DataSchema dataSchema,
*/
private void setSQLGroupByInResultTable(BrokerResponseNative brokerResponseNative, DataSchema dataSchema,
Collection<DataTable> dataTables) {
- DataSchema resultTableSchema = getSQLResultTableSchema(dataSchema);
IndexedTable indexedTable = getIndexedTable(dataSchema, dataTables);
Iterator<Record> sortedIterator = indexedTable.iterator();
+ DataSchema prePostAggregationDataSchema = getPrePostAggregationDataSchema(dataSchema);
int limit = _queryContext.getLimit();
List<Object[]> rows = new ArrayList<>(limit);
+ for (int i = 0; i < limit && sortedIterator.hasNext(); i++) {
+ Object[] row = sortedIterator.next().getValues();
+ for (int j = 0; j < _numAggregationFunctions; j++) {
+ int valueIndex = j + _numGroupByExpressions;
+ row[valueIndex] =
+ AggregationFunctionUtils.getSerializableValue(_aggregationFunctions[j].extractFinalResult(row[valueIndex]));
+ }
+ rows.add(row);
+ }
if (_sqlQuery) {
// SQL query with SQL group-by mode and response format
- // NOTE: For SQL query, need to reorder the columns in the data table based on the select expressions.
-
- int[] selectExpressionIndexMap = getSelectExpressionIndexMap();
- int numSelectExpressions = selectExpressionIndexMap.length;
- String[] columnNames = resultTableSchema.getColumnNames();
- DataSchema.ColumnDataType[] columnDataTypes = resultTableSchema.getColumnDataTypes();
- String[] reorderedColumnNames = new String[numSelectExpressions];
- DataSchema.ColumnDataType[] reorderedColumnDataTypes = new DataSchema.ColumnDataType[numSelectExpressions];
- resultTableSchema = new DataSchema(reorderedColumnNames, reorderedColumnDataTypes);
- for (int i = 0; i < numSelectExpressions; i++) {
- reorderedColumnNames[i] = columnNames[selectExpressionIndexMap[i]];
- reorderedColumnDataTypes[i] = columnDataTypes[selectExpressionIndexMap[i]];
- }
- while (rows.size() < limit && sortedIterator.hasNext()) {
- Record nextRecord = sortedIterator.next();
- Object[] values = nextRecord.getValues();
- for (int i = 0; i < _numAggregationFunctions; i++) {
- int valueIndex = i + _numGroupByExpressions;
- values[valueIndex] = AggregationFunctionUtils
- .getSerializableValue(_aggregationFunctions[i].extractFinalResult(values[valueIndex]));
- }
- Object[] reorderedValues = new Object[numSelectExpressions];
Review comment:
looks like the reordering logic happens within postAggregationHandler, whether we have post aggregations functions or not? Can we add a comment either here or inside postAggregationHandler, that reordering is handled as one of the post aggregation operations?
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
##########
@@ -231,6 +231,13 @@ public void testHardcodedSqlQueries()
testSqlQuery(query, Collections.singletonList(query));
query = "SELECT MAX(ArrDelay), Month FROM mytable GROUP BY Month ORDER BY ABS(Month - 6) + MAX(ArrDelay)";
testSqlQuery(query, Collections.singletonList(query));
+
+ // Post-aggregation in SELECT
Review comment:
how about adding a query with literals
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #5867:
URL: https://github.com/apache/incubator-pinot/pull/5867#discussion_r472409505
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
##########
@@ -160,103 +160,49 @@ public void reduceAndSetResults(String tableName, DataSchema dataSchema,
*/
private void setSQLGroupByInResultTable(BrokerResponseNative brokerResponseNative, DataSchema dataSchema,
Collection<DataTable> dataTables) {
- DataSchema resultTableSchema = getSQLResultTableSchema(dataSchema);
IndexedTable indexedTable = getIndexedTable(dataSchema, dataTables);
Iterator<Record> sortedIterator = indexedTable.iterator();
+ DataSchema prePostAggregationDataSchema = getPrePostAggregationDataSchema(dataSchema);
int limit = _queryContext.getLimit();
List<Object[]> rows = new ArrayList<>(limit);
+ for (int i = 0; i < limit && sortedIterator.hasNext(); i++) {
+ Object[] row = sortedIterator.next().getValues();
+ for (int j = 0; j < _numAggregationFunctions; j++) {
+ int valueIndex = j + _numGroupByExpressions;
+ row[valueIndex] =
+ AggregationFunctionUtils.getSerializableValue(_aggregationFunctions[j].extractFinalResult(row[valueIndex]));
+ }
+ rows.add(row);
+ }
if (_sqlQuery) {
// SQL query with SQL group-by mode and response format
- // NOTE: For SQL query, need to reorder the columns in the data table based on the select expressions.
-
- int[] selectExpressionIndexMap = getSelectExpressionIndexMap();
- int numSelectExpressions = selectExpressionIndexMap.length;
- String[] columnNames = resultTableSchema.getColumnNames();
- DataSchema.ColumnDataType[] columnDataTypes = resultTableSchema.getColumnDataTypes();
- String[] reorderedColumnNames = new String[numSelectExpressions];
- DataSchema.ColumnDataType[] reorderedColumnDataTypes = new DataSchema.ColumnDataType[numSelectExpressions];
- resultTableSchema = new DataSchema(reorderedColumnNames, reorderedColumnDataTypes);
- for (int i = 0; i < numSelectExpressions; i++) {
- reorderedColumnNames[i] = columnNames[selectExpressionIndexMap[i]];
- reorderedColumnDataTypes[i] = columnDataTypes[selectExpressionIndexMap[i]];
- }
- while (rows.size() < limit && sortedIterator.hasNext()) {
- Record nextRecord = sortedIterator.next();
- Object[] values = nextRecord.getValues();
- for (int i = 0; i < _numAggregationFunctions; i++) {
- int valueIndex = i + _numGroupByExpressions;
- values[valueIndex] = AggregationFunctionUtils
- .getSerializableValue(_aggregationFunctions[i].extractFinalResult(values[valueIndex]));
- }
- Object[] reorderedValues = new Object[numSelectExpressions];
Review comment:
Good point. Added comment inside `PostAggregationHandler` stating that the column re-ordering is handled inside.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #5867:
URL: https://github.com/apache/incubator-pinot/pull/5867#discussion_r472403183
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
##########
@@ -231,6 +231,13 @@ public void testHardcodedSqlQueries()
testSqlQuery(query, Collections.singletonList(query));
query = "SELECT MAX(ArrDelay), Month FROM mytable GROUP BY Month ORDER BY ABS(Month - 6) + MAX(ArrDelay)";
testSqlQuery(query, Collections.singletonList(query));
+
+ // Post-aggregation in SELECT
Review comment:
Sounds good, added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #5867:
URL: https://github.com/apache/incubator-pinot/pull/5867
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org