You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/09/23 15:26:36 UTC
[GitHub] [flink] iyupeng opened a new pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
iyupeng opened a new pull request #17344:
URL: https://github.com/apache/flink/pull/17344
## What is the purpose of the change
This pull request follows #14894 which has been inactive for months.
This pull request addresses review comments in the existing pull request and is based on the existing one.
All the code changes are targeted to make it possible to push down local aggregations from computing layer to storage layer for better computing & I/O performance.
This feature is disabled by default.
## Brief change log
- *Add a new configuration of `table.optimizer.source.aggregate-pushdown-enabled` to control the aggregation push down, and the default value is `false` which is up to the users to decide enable it or not.*
- *Add an abstract class of `PushLocalAggIntoScanRuleBase` for supporting both with and without sort operator.*
- *Add a new rule of `PushLocalSortAggWithoutSortIntoScanRule` for sort aggregation without sort case.*
- *Add a new rule of `PushLocalSortAggWithSortIntoScanRule` for sort aggregation with sort case.*
- *Add a new rule of `PushLocalHashAggIntoScanRule` for hash aggregation case.*
- *Add new push down spec `AggregatePushDownSpec` to serialize/deserialize aggregations.*
- *Implement the `SupportsAggregatePushDown` interface for `TestValuesScanTableSource`.*
- *Some other unit tests and integration tests.*
## Verifying this change
This change added tests and can be verified as follows:
- *Added unit test class `PushLocalAggIntoTableSourceScanRuleTest` for plan verifications*
- *Added integration test class `LocalAggregatePushDownITCase` for end-to-end functional test*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493)
* 4e795eef112edd7a5f705596151cacee93d53c03 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r732920524
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
+ rowData.setRowKind(row.getKind());
+ result.add(rowData);
+ }
+ numRetained++;
+ }
+
+ // handle limit. No aggregates will be pushed down when there is a limit.
if (result.size() >= limit) {
return result;
}
- boolean isRetained =
- FilterUtils.isRetainedAfterApplyingFilterPredicates(
- filterPredicates, getValueGetter(row));
- if (isRetained) {
- final Row projectedRow = projectRow(row);
- final RowData rowData = (RowData) converter.toInternal(projectedRow);
- if (rowData != null) {
- if (numRetained >= numElementToSkip) {
- rowData.setRowKind(row.getKind());
- result.add(rowData);
- }
- numRetained++;
- }
+ }
+ }
+
+ return result;
+ }
+
+ private List<Row> applyAggregatesToRows(List<Row> rows) {
+ if (groupingSet != null && groupingSet.length > 0) {
+ // has group by, group firstly
+ Map<Row, List<Row>> buffer = new HashMap<>();
+ for (Row row : rows) {
+ Row bufferKey = new Row(groupingSet.length);
+ for (int i = 0; i < groupingSet.length; i++) {
+ bufferKey.setField(i, row.getField(groupingSet[i]));
+ }
+ if (buffer.containsKey(bufferKey)) {
+ buffer.get(bufferKey).add(row);
+ } else {
+ buffer.put(bufferKey, new ArrayList<>(Collections.singletonList(row)));
}
}
+ List<Row> result = new ArrayList<>();
+ for (Map.Entry<Row, List<Row>> entry : buffer.entrySet()) {
+ result.add(Row.join(entry.getKey(), accumulateRows(entry.getValue())));
+ }
+ return result;
+ } else {
+ return Collections.singletonList(accumulateRows(rows));
+ }
+ }
+
+ // can only apply sum/sum0/avg function for long type fields for testing
+ private Row accumulateRows(List<Row> rows) {
+ Row result = new Row(aggregateExpressions.size());
+ for (int i = 0; i < aggregateExpressions.size(); i++) {
+ FunctionDefinition aggFunction =
+ aggregateExpressions.get(i).getFunctionDefinition();
+ List<FieldReferenceExpression> arguments = aggregateExpressions.get(i).getArgs();
+ if (aggFunction instanceof MinAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row minRow =
+ rows.stream()
+ .min(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, minRow.getField(argIndex));
+ } else if (aggFunction instanceof MaxAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row maxRow =
+ rows.stream()
+ .max(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, maxRow.getField(argIndex));
+ } else if (aggFunction instanceof SumAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum);
+ } else if (aggFunction instanceof Sum0AggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum0 =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum0);
+ } else if (aggFunction instanceof CountAggFunction
Review comment:
if all inputs are null, CountAggFunction should return 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r740206641
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsAggregatePushDown.java
##########
@@ -124,8 +124,6 @@
*
* <p>Regardless if this interface is implemented or not, a final aggregation is always applied in a
* subsequent operation after the source.
- *
- * <p>Note: currently, the {@link SupportsAggregatePushDown} is not supported by planner.
Review comment:
That would be great. The more information in the interface description, the better for all implementers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r738468035
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala
##########
@@ -1002,6 +1001,12 @@ abstract class TableTestUtil(
.getConfiguration
.set(ExecutionOptions.BATCH_SHUFFLE_MODE, BatchShuffleMode.ALL_EXCHANGES_PIPELINED)
+ // Disable push down aggregates to avoid conflicts with existing test cases that verify plans.
+ tableEnv.getConfig
+ .getConfiguration
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, false)
Review comment:
Currently, this pr do not change all tests except you added, so remove it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r740052182
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsAggregatePushDown.java
##########
@@ -124,8 +124,6 @@
*
* <p>Regardless if this interface is implemented or not, a final aggregation is always applied in a
* subsequent operation after the source.
- *
- * <p>Note: currently, the {@link SupportsAggregatePushDown} is not supported by planner.
Review comment:
@iyupeng are we sure that the JavaDocs don't need more updates for this PR?
For example, is this description still correct?
```
* <p>Note: The local aggregate push down strategy is all or nothing, it can only be pushed down if
* all aggregate functions are supported.
```
Also maybe we should mention the config option that was added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735218756
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/FlinkBatchRuleSets.scala
##########
@@ -446,6 +446,9 @@ object FlinkBatchRuleSets {
*/
val PHYSICAL_REWRITE: RuleSet = RuleSets.ofList(
EnforceLocalHashAggRule.INSTANCE,
- EnforceLocalSortAggRule.INSTANCE
+ EnforceLocalSortAggRule.INSTANCE,
+ PushLocalHashAggIntoScanRule.INSTANCE,
+ PushLocalSortAggWithSortIntoScanRule.INSTANCE,
+ PushLocalSortAggWithoutSortIntoScanRule.INSTANCE
Review comment:
added 3 new rules to push down local aggregates with `Calc` when `projection` is not pushed down:
`PushLocalHashAggWithCalcIntoScanRule`, `PushLocalSortAggWithCalcIntoScanRule`, `PushLocalSortAggWithSortAndCalcIntoScanRule`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472)
* 4ad29008075dc93463c07ce098cd7b1086104d7b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa36f90cd26e229c5e0f7dfa66a756d4817007b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469)
* 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-947494587
@iyupeng sorry for the late response, I start to review it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493)
* 4e795eef112edd7a5f705596151cacee93d53c03 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-949094756
@godfreyhe Thanks for your comments. I plan to push new commits in a few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r732918626
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
+ rowData.setRowKind(row.getKind());
+ result.add(rowData);
+ }
+ numRetained++;
+ }
+
+ // handle limit. No aggregates will be pushed down when there is a limit.
if (result.size() >= limit) {
return result;
}
- boolean isRetained =
- FilterUtils.isRetainedAfterApplyingFilterPredicates(
- filterPredicates, getValueGetter(row));
- if (isRetained) {
- final Row projectedRow = projectRow(row);
- final RowData rowData = (RowData) converter.toInternal(projectedRow);
- if (rowData != null) {
- if (numRetained >= numElementToSkip) {
- rowData.setRowKind(row.getKind());
- result.add(rowData);
- }
- numRetained++;
- }
+ }
+ }
+
+ return result;
+ }
+
+ private List<Row> applyAggregatesToRows(List<Row> rows) {
+ if (groupingSet != null && groupingSet.length > 0) {
+ // has group by, group firstly
+ Map<Row, List<Row>> buffer = new HashMap<>();
+ for (Row row : rows) {
+ Row bufferKey = new Row(groupingSet.length);
+ for (int i = 0; i < groupingSet.length; i++) {
+ bufferKey.setField(i, row.getField(groupingSet[i]));
+ }
+ if (buffer.containsKey(bufferKey)) {
+ buffer.get(bufferKey).add(row);
+ } else {
+ buffer.put(bufferKey, new ArrayList<>(Collections.singletonList(row)));
}
}
+ List<Row> result = new ArrayList<>();
+ for (Map.Entry<Row, List<Row>> entry : buffer.entrySet()) {
+ result.add(Row.join(entry.getKey(), accumulateRows(entry.getValue())));
+ }
+ return result;
+ } else {
+ return Collections.singletonList(accumulateRows(rows));
+ }
+ }
+
+ // can only apply sum/sum0/avg function for long type fields for testing
+ private Row accumulateRows(List<Row> rows) {
+ Row result = new Row(aggregateExpressions.size());
+ for (int i = 0; i < aggregateExpressions.size(); i++) {
+ FunctionDefinition aggFunction =
+ aggregateExpressions.get(i).getFunctionDefinition();
+ List<FieldReferenceExpression> arguments = aggregateExpressions.get(i).getArgs();
+ if (aggFunction instanceof MinAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row minRow =
+ rows.stream()
+ .min(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, minRow.getField(argIndex));
+ } else if (aggFunction instanceof MaxAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row maxRow =
+ rows.stream()
+ .max(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, maxRow.getField(argIndex));
+ } else if (aggFunction instanceof SumAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
Review comment:
if all inputs are null, SumAggFunction should output null
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
+ rowData.setRowKind(row.getKind());
+ result.add(rowData);
+ }
+ numRetained++;
+ }
+
+ // handle limit. No aggregates will be pushed down when there is a limit.
if (result.size() >= limit) {
return result;
}
- boolean isRetained =
- FilterUtils.isRetainedAfterApplyingFilterPredicates(
- filterPredicates, getValueGetter(row));
- if (isRetained) {
- final Row projectedRow = projectRow(row);
- final RowData rowData = (RowData) converter.toInternal(projectedRow);
- if (rowData != null) {
- if (numRetained >= numElementToSkip) {
- rowData.setRowKind(row.getKind());
- result.add(rowData);
- }
- numRetained++;
- }
+ }
+ }
+
+ return result;
+ }
+
+ private List<Row> applyAggregatesToRows(List<Row> rows) {
+ if (groupingSet != null && groupingSet.length > 0) {
+ // has group by, group firstly
+ Map<Row, List<Row>> buffer = new HashMap<>();
+ for (Row row : rows) {
+ Row bufferKey = new Row(groupingSet.length);
+ for (int i = 0; i < groupingSet.length; i++) {
+ bufferKey.setField(i, row.getField(groupingSet[i]));
+ }
+ if (buffer.containsKey(bufferKey)) {
+ buffer.get(bufferKey).add(row);
+ } else {
+ buffer.put(bufferKey, new ArrayList<>(Collections.singletonList(row)));
}
}
+ List<Row> result = new ArrayList<>();
+ for (Map.Entry<Row, List<Row>> entry : buffer.entrySet()) {
+ result.add(Row.join(entry.getKey(), accumulateRows(entry.getValue())));
+ }
+ return result;
+ } else {
+ return Collections.singletonList(accumulateRows(rows));
+ }
+ }
+
+ // can only apply sum/sum0/avg function for long type fields for testing
+ private Row accumulateRows(List<Row> rows) {
+ Row result = new Row(aggregateExpressions.size());
+ for (int i = 0; i < aggregateExpressions.size(); i++) {
+ FunctionDefinition aggFunction =
+ aggregateExpressions.get(i).getFunctionDefinition();
+ List<FieldReferenceExpression> arguments = aggregateExpressions.get(i).getArgs();
+ if (aggFunction instanceof MinAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row minRow =
+ rows.stream()
+ .min(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, minRow.getField(argIndex));
+ } else if (aggFunction instanceof MaxAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row maxRow =
+ rows.stream()
+ .max(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, maxRow.getField(argIndex));
+ } else if (aggFunction instanceof SumAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum);
+ } else if (aggFunction instanceof Sum0AggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum0 =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum0);
+ } else if (aggFunction instanceof CountAggFunction
Review comment:
if all inputs are null, CountAggFunction should output 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454)
* 0fa36f90cd26e229c5e0f7dfa66a756d4817007b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735219717
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | name,
+ | d,
+ | p,
+ | count(*)
+ |FROM (
+ | SELECT
+ | name,
+ | sum(deposit) as d,
+ | max(points) as p
+ | FROM AggregatableTable
+ | GROUP BY name
+ |) t
+ |GROUP BY name, d, p
+ |""".stripMargin,
+ Seq(
+ row("tom", 200, 1000, 1),
+ row("mary", 100, 1000, 1),
+ row("jack", 150, 1300, 1),
+ row("rose", 100, 500, 1),
+ row("danny", 300, 300, 1),
+ row("tommas", 400, 4000, 1),
+ row("olivia", 50, 9000, 1),
+ row("stef", 100, 1900, 1),
+ row("emma", 180, 800, 1),
+ row("benji", 170, 11000, 1),
+ row("eva", 200, 1000, 1))
+ )
+ }
+
+ @Test
+ def testPushDownLocalAggAfterFilterPushDown(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |WHERE age <= 20
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20))
+ )
+ }
+
+ @Test
+ def testLocalAggWithLimit(): Unit = {
Review comment:
removed unsupported test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r732638497
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalHashAggIntoScanRule.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalHashAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local hash or sort aggregate which without sort into a {@link
Review comment:
the comment should be updated, remove "or sort aggregate"
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalSortAggWithoutSortIntoScanRule.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalSortAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local hash or sort aggregate which without sort into a {@link
+ * BatchPhysicalTableSourceScan} whose table is a {@link TableSourceTable} with a source supporting
+ * {@link SupportsAggregatePushDown}. The {@link
+ * OptimizerConfigOptions#TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED} need to be true.
+ *
+ * <p>Suppose we have the original physical plan:
+ *
+ * <pre>{@code
+ * BatchPhysicalHashAggregate (global)
Review comment:
BatchPhysicalSortAggregate
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean isMatch(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets = Arrays.asList(localAgg.grouping(), localAgg.auxGrouping());
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType, groupingSets, aggCallList, producedType, newTableSource);
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable);
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ newFlinkStatistic,
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(localAgg.getRowType());
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(TableSourceTable tableSourceTable) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+ if (oldStatistic == FlinkStatistic.UNKNOWN()) {
+ return oldStatistic;
+ } else {
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder().statistic(oldStatistic).tableStats(null).build();
Review comment:
unique key info in `FlinkStatistic` may be also changed
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/AggregateUtil.scala
##########
@@ -279,6 +281,21 @@ object AggregateUtil extends Enumeration {
isBounded = false)
}
+ def deriveSumAndCountFromAvg(avgAggFunction: UserDefinedFunction
Review comment:
use specific type:`AvgAggFunction` instead of `UserDefinedFunction` here
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
Review comment:
port this class into JAVA
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | name,
+ | d,
+ | p,
+ | count(*)
+ |FROM (
+ | SELECT
+ | name,
+ | sum(deposit) as d,
+ | max(points) as p
+ | FROM AggregatableTable
+ | GROUP BY name
+ |) t
+ |GROUP BY name, d, p
+ |""".stripMargin,
+ Seq(
+ row("tom", 200, 1000, 1),
+ row("mary", 100, 1000, 1),
+ row("jack", 150, 1300, 1),
+ row("rose", 100, 500, 1),
+ row("danny", 300, 300, 1),
+ row("tommas", 400, 4000, 1),
+ row("olivia", 50, 9000, 1),
+ row("stef", 100, 1900, 1),
+ row("emma", 180, 800, 1),
+ row("benji", 170, 11000, 1),
+ row("eva", 200, 1000, 1))
+ )
+ }
+
+ @Test
+ def testPushDownLocalAggAfterFilterPushDown(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |WHERE age <= 20
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20))
+ )
+ }
+
+ @Test
+ def testLocalAggWithLimit(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | (
+ | SELECT * FROM AggregatableTable
+ | LIMIT 10
+ | ) t
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(107, 430, 4, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testLocalAggWithUDAF(): Unit = {
+ // add UDAF
+ tEnv.createTemporarySystemFunction(
+ "udaf_collect",
+ new CollectAggFunction(DataTypes.BIGINT().getLogicalType))
+
+ checkResult(
+ """
+ |SELECT
+ | udaf_collect(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row("{100=1}", 1, "m", 34),
+ row("{100=2, 180=1}", 3, "f", 25),
+ row("{170=1}", 1, "m", 28),
+ row("{200=1}", 1, "f", 20),
+ row("{300=1}", 1, "m", 27),
+ row("{400=1, 150=1, 200=1}", 3, "m", 23),
+ row("{50=1}", 1, "f", 19))
+ )
+ }
+
+ @Test
+ def testLocalAggWithUnsupportedDataTypes(): Unit = {
+ // only agg on Long columns and count are supported to be pushed down
+ // in {@link TestValuesTableFactory}
+
+ checkResult(
+ """
+ |SELECT
+ | min(age),
+ | max(height),
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(19, 172, 50, 50, 1, "f", 19),
+ row(20, 180, 200, 200, 1, "f", 20),
+ row(23, 182, 250, 750, 3, "m", 23),
+ row(25, 171, 126, 380, 3, "f", 25),
+ row(27, 175, 300, 300, 1, "m", 27),
+ row(28, 165, 170, 170, 1, "m", 28),
+ row(34, 170, 100, 100, 1, "m", 34))
+ )
+ }
+
+ @Test
+ def testLocalAggWithColumnExpression1(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit + points),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(250, 7050, 3, "m", 23),
+ row(126, 2680, 3, "f", 25),
+ row(300, 600, 1, "m", 27),
+ row(50, 9050, 1, "f", 19),
+ row(100, 2000, 1, "m", 34),
+ row(170, 11170, 1, "m", 28),
+ row(200, 1200, 1, "f", 20))
+ )
+ }
+
+ @Test
+ def testLocalAggWithColumnExpression2(): Unit = {
Review comment:
testLocalAggWithColumnExpression2 -> testLocalAggWithDistinct
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | name,
+ | d,
+ | p,
+ | count(*)
+ |FROM (
+ | SELECT
+ | name,
+ | sum(deposit) as d,
+ | max(points) as p
+ | FROM AggregatableTable
+ | GROUP BY name
+ |) t
+ |GROUP BY name, d, p
+ |""".stripMargin,
+ Seq(
+ row("tom", 200, 1000, 1),
+ row("mary", 100, 1000, 1),
+ row("jack", 150, 1300, 1),
+ row("rose", 100, 500, 1),
+ row("danny", 300, 300, 1),
+ row("tommas", 400, 4000, 1),
+ row("olivia", 50, 9000, 1),
+ row("stef", 100, 1900, 1),
+ row("emma", 180, 800, 1),
+ row("benji", 170, 11000, 1),
+ row("eva", 200, 1000, 1))
+ )
+ }
+
+ @Test
+ def testPushDownLocalAggAfterFilterPushDown(): Unit = {
+ checkResult(
Review comment:
the local aggregate can not be pushed down to source, since the filter can not be pushed down the source.
see the plan:
```
Sink(table=[default_catalog.default_database.Unregistered_Collect_Sink_1], fields=[EXPR$0, EXPR$1, EXPR$2, gender, age])
+- Calc(select=[EXPR$0, EXPR$1, EXPR$2, gender, age])
+- HashAggregate(isMerge=[true], groupBy=[gender, age], select=[gender, age, Final_AVG(sum$0, count$1) AS EXPR$0, Final_SUM(sum$2) AS EXPR$1, Final_COUNT(count1$3) AS EXPR$2])
+- Exchange(distribution=[hash[gender, age]])
+- LocalHashAggregate(groupBy=[gender, age], select=[gender, age, Partial_AVG(deposit) AS (sum$0, count$1), Partial_SUM(deposit) AS sum$2, Partial_COUNT(*) AS count1$3])
+- Calc(select=[gender, age, deposit], where=[<=(age, 20)])
+- TableSourceScan(table=[[default_catalog, default_database, AggregatableTable, filter=[], project=[age, gender, deposit], metadata=[]]], fields=[age, gender, deposit])
```
please add `filterable-fields` into `with` clause and then the filter can be pushed down successfully. see PushFilterInCalcIntoTableSourceRuleTest#L79
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
Review comment:
ditto
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -131,4 +130,23 @@ class TableSourceTable(
flinkContext,
abilitySpecs ++ newAbilitySpecs)
}
+
+ /**
+ * Creates a copy of this table, changing the rowType
+ *
+ * @param newRowType new row type
+ * @return New TableSourceTable instance with new row type
+ */
+ def copy(newRowType: RelDataType): TableSourceTable = {
Review comment:
providing the `copy` method with `FlinkStatistic` is more useful
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
Review comment:
this is unnecessary
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
+ rowData.setRowKind(row.getKind());
+ result.add(rowData);
+ }
+ numRetained++;
+ }
+
+ // handle limit. No aggregates will be pushed down when there is a limit.
if (result.size() >= limit) {
return result;
}
- boolean isRetained =
- FilterUtils.isRetainedAfterApplyingFilterPredicates(
- filterPredicates, getValueGetter(row));
- if (isRetained) {
- final Row projectedRow = projectRow(row);
- final RowData rowData = (RowData) converter.toInternal(projectedRow);
- if (rowData != null) {
- if (numRetained >= numElementToSkip) {
- rowData.setRowKind(row.getKind());
- result.add(rowData);
- }
- numRetained++;
- }
+ }
+ }
+
+ return result;
+ }
+
+ private List<Row> applyAggregatesToRows(List<Row> rows) {
+ if (groupingSet != null && groupingSet.length > 0) {
+ // has group by, group firstly
+ Map<Row, List<Row>> buffer = new HashMap<>();
+ for (Row row : rows) {
+ Row bufferKey = new Row(groupingSet.length);
+ for (int i = 0; i < groupingSet.length; i++) {
+ bufferKey.setField(i, row.getField(groupingSet[i]));
+ }
+ if (buffer.containsKey(bufferKey)) {
+ buffer.get(bufferKey).add(row);
+ } else {
+ buffer.put(bufferKey, new ArrayList<>(Collections.singletonList(row)));
}
}
+ List<Row> result = new ArrayList<>();
+ for (Map.Entry<Row, List<Row>> entry : buffer.entrySet()) {
+ result.add(Row.join(entry.getKey(), accumulateRows(entry.getValue())));
+ }
+ return result;
+ } else {
+ return Collections.singletonList(accumulateRows(rows));
+ }
+ }
+
+ // can only apply sum/sum0/avg function for long type fields for testing
+ private Row accumulateRows(List<Row> rows) {
+ Row result = new Row(aggregateExpressions.size());
+ for (int i = 0; i < aggregateExpressions.size(); i++) {
+ FunctionDefinition aggFunction =
+ aggregateExpressions.get(i).getFunctionDefinition();
+ List<FieldReferenceExpression> arguments = aggregateExpressions.get(i).getArgs();
+ if (aggFunction instanceof MinAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row minRow =
+ rows.stream()
+ .min(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, minRow.getField(argIndex));
+ } else if (aggFunction instanceof MaxAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row maxRow =
+ rows.stream()
+ .max(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, maxRow.getField(argIndex));
+ } else if (aggFunction instanceof SumAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum);
+ } else if (aggFunction instanceof Sum0AggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum0 =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
+ result.setField(i, finalSum0);
+ } else if (aggFunction instanceof CountAggFunction
Review comment:
if all input is nll, CountAggFunction will return 0
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalSortAggWithoutSortIntoScanRule.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalSortAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local hash or sort aggregate which without sort into a {@link
Review comment:
please update the java doc: remove "local hash or"
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
Review comment:
nit: WithOut -> without
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalSortAggWithoutSortIntoScanRule.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalSortAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local hash or sort aggregate which without sort into a {@link
+ * BatchPhysicalTableSourceScan} whose table is a {@link TableSourceTable} with a source supporting
+ * {@link SupportsAggregatePushDown}. The {@link
+ * OptimizerConfigOptions#TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED} need to be true.
+ *
+ * <p>Suppose we have the original physical plan:
+ *
+ * <pre>{@code
+ * BatchPhysicalHashAggregate (global)
+ * +- BatchPhysicalExchange (hash by group keys if group keys is not empty, else singleton)
+ * +- BatchPhysicalLocalSortAggregate (local)
+ * +- BatchPhysicalTableSourceScan
+ * }</pre>
+ *
+ * <p>This physical plan will be rewritten to:
+ *
+ * <pre>{@code
+ * BatchPhysicalHashAggregate (global)
Review comment:
BatchPhysicalSortAggregate
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchPhysicalTableSourceScan.scala
##########
@@ -18,6 +18,12 @@
package org.apache.flink.table.planner.plan.nodes.physical.batch
+import java.util
+
+import org.apache.calcite.plan._
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.hint.RelHint
+import org.apache.calcite.rel.metadata.RelMetadataQuery
Review comment:
nit: reorder the imports
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/AggregatePushDownSpec.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.source;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.expressions.AggregateExpression;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
+import org.apache.flink.table.planner.functions.aggfunctions.AvgAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.CountAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.Sum0AggFunction;
+import org.apache.flink.table.planner.plan.utils.AggregateInfo;
+import org.apache.flink.table.planner.plan.utils.AggregateInfoList;
+import org.apache.flink.table.planner.plan.utils.AggregateUtil;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A sub-class of {@link SourceAbilitySpec} that can not only serialize/deserialize the aggregation
+ * to/from JSON, but also can push the local aggregate into a {@link SupportsAggregatePushDown}.
+ */
+@JsonTypeName("AggregatePushDown")
+public class AggregatePushDownSpec extends SourceAbilitySpecBase {
+
+ public static final String FIELD_NAME_INPUT_TYPE = "inputType";
+
+ public static final String FIELD_NAME_GROUPING_SETS = "groupingSets";
+
+ public static final String FIELD_NAME_AGGREGATE_CALLS = "aggregateCalls";
+
+ @JsonProperty(FIELD_NAME_INPUT_TYPE)
+ private final RowType inputType;
+
+ @JsonProperty(FIELD_NAME_GROUPING_SETS)
+ private final List<int[]> groupingSets;
+
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS)
+ private final List<AggregateCall> aggregateCalls;
+
+ @JsonCreator
+ public AggregatePushDownSpec(
+ @JsonProperty(FIELD_NAME_INPUT_TYPE) RowType inputType,
+ @JsonProperty(FIELD_NAME_GROUPING_SETS) List<int[]> groupingSets,
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS) List<AggregateCall> aggregateCalls,
+ @JsonProperty(FIELD_NAME_PRODUCED_TYPE) RowType producedType) {
+ super(producedType);
+
+ this.inputType = inputType;
+ this.groupingSets = new ArrayList<>(checkNotNull(groupingSets));
+ this.aggregateCalls = aggregateCalls;
+ }
+
+ @Override
+ public void apply(DynamicTableSource tableSource, SourceAbilityContext context) {
+ checkArgument(getProducedType().isPresent());
+ apply(inputType, groupingSets, aggregateCalls, getProducedType().get(), tableSource);
+ }
+
+ @Override
+ public String getDigests(SourceAbilityContext context) {
+ int[] grouping = ArrayUtils.addAll(groupingSets.get(0), groupingSets.get(1));
+ String groupingStr =
+ Arrays.stream(grouping)
+ .mapToObj(index -> inputType.getFieldNames().get(index))
+ .collect(Collectors.joining(","));
+
+ List<AggregateExpression> aggregateExpressions =
+ buildAggregateExpressions(inputType, aggregateCalls);
+ String aggFunctionsStr =
+ aggregateExpressions.stream()
+ .map(AggregateExpression::asSummaryString)
+ .collect(Collectors.joining(","));
+
+ return "aggregates=[grouping=["
+ + groupingStr
+ + "], aggFunctions=["
+ + aggFunctionsStr
+ + "]]";
+ }
+
+ public static boolean apply(
+ RowType inputType,
+ List<int[]> groupingSets,
+ List<AggregateCall> aggregateCalls,
+ RowType producedType,
+ DynamicTableSource tableSource) {
+ List<AggregateExpression> aggregateExpressions =
+ buildAggregateExpressions(inputType, aggregateCalls);
+
+ if (tableSource instanceof SupportsAggregatePushDown) {
+ DataType producedDataType = TypeConversions.fromLogicalToDataType(producedType);
+ return ((SupportsAggregatePushDown) tableSource)
+ .applyAggregates(groupingSets, aggregateExpressions, producedDataType);
+ } else {
+ throw new TableException(
+ String.format(
+ "%s does not support SupportsAggregatePushDown.",
+ tableSource.getClass().getName()));
+ }
+ }
+
+ private static List<AggregateExpression> buildAggregateExpressions(
+ RowType inputType, List<AggregateCall> aggregateCalls) {
+ AggregateInfoList aggInfoList =
+ AggregateUtil.transformToBatchAggregateInfoList(
Review comment:
we should improve the java-doc or add some validation to make sure the AggregatePushDownSpec can be only used for batch now
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean isMatch(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets = Arrays.asList(localAgg.grouping(), localAgg.auxGrouping());
Review comment:
`grouping` and `auxGrouping` should be combined into ONE array. Actually, we only support 'single' group aggregate. `groupingSets` here should always have one element.
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/FlinkBatchRuleSets.scala
##########
@@ -446,6 +446,9 @@ object FlinkBatchRuleSets {
*/
val PHYSICAL_REWRITE: RuleSet = RuleSets.ofList(
EnforceLocalHashAggRule.INSTANCE,
- EnforceLocalSortAggRule.INSTANCE
+ EnforceLocalSortAggRule.INSTANCE,
+ PushLocalHashAggIntoScanRule.INSTANCE,
+ PushLocalSortAggWithSortIntoScanRule.INSTANCE,
+ PushLocalSortAggWithoutSortIntoScanRule.INSTANCE
Review comment:
Some table sources may do not implement `SupportsProjectionPushDown`, so the pattern `Scan -> Calc (only reference projection) -> local agg` does also exist
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -18,19 +18,18 @@
package org.apache.flink.table.planner.plan.schema
+import java.util
+
+import com.google.common.collect.ImmutableList
+import org.apache.calcite.plan.RelOptSchema
+import org.apache.calcite.rel.`type`.RelDataType
Review comment:
reorder the imports
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/AggregateUtil.scala
##########
@@ -279,6 +281,21 @@ object AggregateUtil extends Enumeration {
isBounded = false)
}
+ def deriveSumAndCountFromAvg(avgAggFunction: UserDefinedFunction
+ ): (Sum0AggFunction, CountAggFunction) = {
Review comment:
the indent seems not correct
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/AggregatePushDownSpec.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.source;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.expressions.AggregateExpression;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
+import org.apache.flink.table.planner.functions.aggfunctions.AvgAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.CountAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.Sum0AggFunction;
+import org.apache.flink.table.planner.plan.utils.AggregateInfo;
+import org.apache.flink.table.planner.plan.utils.AggregateInfoList;
+import org.apache.flink.table.planner.plan.utils.AggregateUtil;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A sub-class of {@link SourceAbilitySpec} that can not only serialize/deserialize the aggregation
+ * to/from JSON, but also can push the local aggregate into a {@link SupportsAggregatePushDown}.
+ */
+@JsonTypeName("AggregatePushDown")
+public class AggregatePushDownSpec extends SourceAbilitySpecBase {
+
+ public static final String FIELD_NAME_INPUT_TYPE = "inputType";
+
+ public static final String FIELD_NAME_GROUPING_SETS = "groupingSets";
+
+ public static final String FIELD_NAME_AGGREGATE_CALLS = "aggregateCalls";
+
+ @JsonProperty(FIELD_NAME_INPUT_TYPE)
+ private final RowType inputType;
+
+ @JsonProperty(FIELD_NAME_GROUPING_SETS)
+ private final List<int[]> groupingSets;
+
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS)
+ private final List<AggregateCall> aggregateCalls;
+
+ @JsonCreator
+ public AggregatePushDownSpec(
+ @JsonProperty(FIELD_NAME_INPUT_TYPE) RowType inputType,
+ @JsonProperty(FIELD_NAME_GROUPING_SETS) List<int[]> groupingSets,
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS) List<AggregateCall> aggregateCalls,
+ @JsonProperty(FIELD_NAME_PRODUCED_TYPE) RowType producedType) {
+ super(producedType);
+
+ this.inputType = inputType;
+ this.groupingSets = new ArrayList<>(checkNotNull(groupingSets));
+ this.aggregateCalls = aggregateCalls;
+ }
+
+ @Override
+ public void apply(DynamicTableSource tableSource, SourceAbilityContext context) {
+ checkArgument(getProducedType().isPresent());
+ apply(inputType, groupingSets, aggregateCalls, getProducedType().get(), tableSource);
+ }
+
+ @Override
+ public String getDigests(SourceAbilityContext context) {
+ int[] grouping = ArrayUtils.addAll(groupingSets.get(0), groupingSets.get(1));
Review comment:
should only use the first element
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | name,
+ | d,
+ | p,
+ | count(*)
+ |FROM (
+ | SELECT
+ | name,
+ | sum(deposit) as d,
+ | max(points) as p
+ | FROM AggregatableTable
+ | GROUP BY name
+ |) t
+ |GROUP BY name, d, p
+ |""".stripMargin,
+ Seq(
+ row("tom", 200, 1000, 1),
+ row("mary", 100, 1000, 1),
+ row("jack", 150, 1300, 1),
+ row("rose", 100, 500, 1),
+ row("danny", 300, 300, 1),
+ row("tommas", 400, 4000, 1),
+ row("olivia", 50, 9000, 1),
+ row("stef", 100, 1900, 1),
+ row("emma", 180, 800, 1),
+ row("benji", 170, 11000, 1),
+ row("eva", 200, 1000, 1))
+ )
+ }
+
+ @Test
+ def testPushDownLocalAggAfterFilterPushDown(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |WHERE age <= 20
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20))
+ )
+ }
+
+ @Test
+ def testLocalAggWithLimit(): Unit = {
Review comment:
the unsupported cases can be removed from IT case, because plan tests have covered such case
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
+ rowData.setRowKind(row.getKind());
+ result.add(rowData);
+ }
+ numRetained++;
+ }
+
+ // handle limit. No aggregates will be pushed down when there is a limit.
if (result.size() >= limit) {
return result;
}
- boolean isRetained =
- FilterUtils.isRetainedAfterApplyingFilterPredicates(
- filterPredicates, getValueGetter(row));
- if (isRetained) {
- final Row projectedRow = projectRow(row);
- final RowData rowData = (RowData) converter.toInternal(projectedRow);
- if (rowData != null) {
- if (numRetained >= numElementToSkip) {
- rowData.setRowKind(row.getKind());
- result.add(rowData);
- }
- numRetained++;
- }
+ }
+ }
+
+ return result;
+ }
+
+ private List<Row> applyAggregatesToRows(List<Row> rows) {
+ if (groupingSet != null && groupingSet.length > 0) {
+ // has group by, group firstly
+ Map<Row, List<Row>> buffer = new HashMap<>();
+ for (Row row : rows) {
+ Row bufferKey = new Row(groupingSet.length);
+ for (int i = 0; i < groupingSet.length; i++) {
+ bufferKey.setField(i, row.getField(groupingSet[i]));
+ }
+ if (buffer.containsKey(bufferKey)) {
+ buffer.get(bufferKey).add(row);
+ } else {
+ buffer.put(bufferKey, new ArrayList<>(Collections.singletonList(row)));
}
}
+ List<Row> result = new ArrayList<>();
+ for (Map.Entry<Row, List<Row>> entry : buffer.entrySet()) {
+ result.add(Row.join(entry.getKey(), accumulateRows(entry.getValue())));
+ }
+ return result;
+ } else {
+ return Collections.singletonList(accumulateRows(rows));
+ }
+ }
+
+ // can only apply sum/sum0/avg function for long type fields for testing
+ private Row accumulateRows(List<Row> rows) {
+ Row result = new Row(aggregateExpressions.size());
+ for (int i = 0; i < aggregateExpressions.size(); i++) {
+ FunctionDefinition aggFunction =
+ aggregateExpressions.get(i).getFunctionDefinition();
+ List<FieldReferenceExpression> arguments = aggregateExpressions.get(i).getArgs();
+ if (aggFunction instanceof MinAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row minRow =
+ rows.stream()
+ .min(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, minRow.getField(argIndex));
+ } else if (aggFunction instanceof MaxAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Row maxRow =
+ rows.stream()
+ .max(Comparator.comparing(row -> row.getFieldAs(argIndex)))
+ .get();
+ result.setField(i, maxRow.getField(argIndex));
+ } else if (aggFunction instanceof SumAggFunction) {
+ int argIndex = arguments.get(0).getFieldIndex();
+ Object finalSum =
+ rows.stream()
+ .filter(row -> row.getField(argIndex) != null)
+ .mapToLong(row -> row.getFieldAs(argIndex))
+ .sum();
Review comment:
if all input is null, SumAggFunction will output null
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
Review comment:
Please add some test cases to cover push down partition and local aggregate , push down meta column with local aggregate, push down local aggregate without projection push down. so does IT case
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
+ protected BatchTableTestUtil util = batchTestUtil(new TableConfig());
+
+ @Before
+ public void setup() {
+ TableConfig tableConfig = util.tableEnv().getConfig();
+ tableConfig
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ String ddl =
+ "CREATE TABLE inventory (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testDisablePushDownLocalAgg() {
+ // disable push down local agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false);
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithoutGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithoutSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithAuxGrouping() {
Review comment:
the test is not correct
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
Review comment:
skipping the original data instead of the result data is more reasonable
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
+ protected BatchTableTestUtil util = batchTestUtil(new TableConfig());
+
+ @Before
+ public void setup() {
+ TableConfig tableConfig = util.tableEnv().getConfig();
+ tableConfig
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ String ddl =
+ "CREATE TABLE inventory (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testDisablePushDownLocalAgg() {
+ // disable push down local agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false);
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithoutGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithoutSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithAuxGrouping() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " name,\n"
+ + " a,\n"
+ + " p,\n"
+ + " count(*)\n"
+ + "FROM (\n"
+ + " SELECT\n"
+ + " name,\n"
+ + " sum(amount) as a,\n"
+ + " max(price) as p\n"
+ + " FROM inventory\n"
+ + " group by name\n"
+ + ") t\n"
+ + " group by name, a, p");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggAfterFilterPushDown() {
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " where id = 123\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCannotPushDownLocalAggAfterLimitPushDown() {
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM (\n"
+ + " SELECT\n"
+ + " *\n"
+ + " FROM inventory\n"
+ + " LIMIT 100\n"
+ + ") t\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCannotPushDownLocalAggWithUDAF() {
+ // add udf
+ util.addTemporarySystemFunction(
+ "udaf_collect", new CollectAggFunction<>(DataTypes.BIGINT().getLogicalType()));
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " udaf_collect(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCannotPushDownLocalAggWithUnsupportedDataTypes() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " max(name),\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by type");
+ }
+
+ @Test
+ public void testCannotPushDownWithColumnExpression() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(amount + price),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " count(id),\n"
+ + " name\n"
+ + "FROM inventory\n"
+ + " group by name");
+ }
+
+ @Test
+ public void testCannotPushDownWithUnsupportedAggFunction() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " count(distinct id),\n"
+ + " name\n"
+ + "FROM inventory\n"
+ + " group by name");
+ }
+
+ @Test
+ public void testCannotPushDownWithWindowAggFunction() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " id,\n"
+ + " amount,\n"
+ + " sum(price) over (partition by name),\n"
+ + " name\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCannotPushDownWithFilter() {
Review comment:
testCannotPushDownWithFilter -> testCannotPushDownWithArgFilter
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
Review comment:
the push down local aggregate has no auxGrouping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925922702
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 (Thu Sep 23 15:29:59 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20895).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735217760
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/AggregatePushDownSpec.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.source;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.expressions.AggregateExpression;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
+import org.apache.flink.table.planner.functions.aggfunctions.AvgAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.CountAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.Sum0AggFunction;
+import org.apache.flink.table.planner.plan.utils.AggregateInfo;
+import org.apache.flink.table.planner.plan.utils.AggregateInfoList;
+import org.apache.flink.table.planner.plan.utils.AggregateUtil;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A sub-class of {@link SourceAbilitySpec} that can not only serialize/deserialize the aggregation
+ * to/from JSON, but also can push the local aggregate into a {@link SupportsAggregatePushDown}.
+ */
+@JsonTypeName("AggregatePushDown")
+public class AggregatePushDownSpec extends SourceAbilitySpecBase {
+
+ public static final String FIELD_NAME_INPUT_TYPE = "inputType";
+
+ public static final String FIELD_NAME_GROUPING_SETS = "groupingSets";
+
+ public static final String FIELD_NAME_AGGREGATE_CALLS = "aggregateCalls";
+
+ @JsonProperty(FIELD_NAME_INPUT_TYPE)
+ private final RowType inputType;
+
+ @JsonProperty(FIELD_NAME_GROUPING_SETS)
+ private final List<int[]> groupingSets;
+
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS)
+ private final List<AggregateCall> aggregateCalls;
+
+ @JsonCreator
+ public AggregatePushDownSpec(
+ @JsonProperty(FIELD_NAME_INPUT_TYPE) RowType inputType,
+ @JsonProperty(FIELD_NAME_GROUPING_SETS) List<int[]> groupingSets,
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS) List<AggregateCall> aggregateCalls,
+ @JsonProperty(FIELD_NAME_PRODUCED_TYPE) RowType producedType) {
+ super(producedType);
+
+ this.inputType = inputType;
+ this.groupingSets = new ArrayList<>(checkNotNull(groupingSets));
+ this.aggregateCalls = aggregateCalls;
+ }
+
+ @Override
+ public void apply(DynamicTableSource tableSource, SourceAbilityContext context) {
+ checkArgument(getProducedType().isPresent());
+ apply(inputType, groupingSets, aggregateCalls, getProducedType().get(), tableSource);
+ }
+
+ @Override
+ public String getDigests(SourceAbilityContext context) {
+ int[] grouping = ArrayUtils.addAll(groupingSets.get(0), groupingSets.get(1));
+ String groupingStr =
+ Arrays.stream(grouping)
+ .mapToObj(index -> inputType.getFieldNames().get(index))
+ .collect(Collectors.joining(","));
+
+ List<AggregateExpression> aggregateExpressions =
+ buildAggregateExpressions(inputType, aggregateCalls);
+ String aggFunctionsStr =
+ aggregateExpressions.stream()
+ .map(AggregateExpression::asSummaryString)
+ .collect(Collectors.joining(","));
+
+ return "aggregates=[grouping=["
+ + groupingStr
+ + "], aggFunctions=["
+ + aggFunctionsStr
+ + "]]";
+ }
+
+ public static boolean apply(
+ RowType inputType,
+ List<int[]> groupingSets,
+ List<AggregateCall> aggregateCalls,
+ RowType producedType,
+ DynamicTableSource tableSource) {
+ List<AggregateExpression> aggregateExpressions =
+ buildAggregateExpressions(inputType, aggregateCalls);
+
+ if (tableSource instanceof SupportsAggregatePushDown) {
+ DataType producedDataType = TypeConversions.fromLogicalToDataType(producedType);
+ return ((SupportsAggregatePushDown) tableSource)
+ .applyAggregates(groupingSets, aggregateExpressions, producedDataType);
+ } else {
+ throw new TableException(
+ String.format(
+ "%s does not support SupportsAggregatePushDown.",
+ tableSource.getClass().getName()));
+ }
+ }
+
+ private static List<AggregateExpression> buildAggregateExpressions(
+ RowType inputType, List<AggregateCall> aggregateCalls) {
+ AggregateInfoList aggInfoList =
+ AggregateUtil.transformToBatchAggregateInfoList(
Review comment:
added an assertion in `apply` function to check `batch` mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735218053
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
Review comment:
moved into `java` as `LocalAggregatePushDownITCase.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735221462
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.scala
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.batch.sql.agg
+
+import org.apache.flink.table.api.DataTypes
+import org.apache.flink.table.api.config.{ExecutionConfigOptions, OptimizerConfigOptions}
+import org.apache.flink.table.planner.factories.TestValuesTableFactory
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase.row
+import org.apache.flink.table.planner.runtime.utils.{BatchTestBase, TestData}
+import org.junit.{Before, Test}
+
+class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ override def before(): Unit = {
+ super.before()
+ env.setParallelism(1) // set sink parallelism to 1
+ conf.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, true)
+ val testDataId = TestValuesTableFactory.registerData(TestData.personData)
+ val ddl =
+ s"""
+ |CREATE TABLE AggregatableTable (
+ | id int,
+ | age int,
+ | name string,
+ | height int,
+ | gender string,
+ | deposit bigint,
+ | points bigint
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$testDataId',
+ | 'filterable-fields' = 'id',
+ | 'bounded' = 'true'
+ |)
+ """.stripMargin
+ tEnv.executeSql(ddl)
+
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+ }
+
+ @Test
+ def testDisablePushDownLocalAgg(): Unit = {
+ // disable push down local agg
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false)
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit) as avg_dep,
+ | sum(deposit),
+ | count(1),
+ | gender
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender
+ |ORDER BY avg_dep
+ |""".stripMargin,
+ Seq(
+ row(126, 630, 5, "f"),
+ row(220, 1320, 6, "m"))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true)
+ }
+
+ @Test
+ def testPushDownLocalHashAggWithOutGroup(): Unit = {
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithoutSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(*)
+ |FROM
+ | AggregatableTable
+ |""".stripMargin,
+ Seq(
+ row(177, 1950, 11))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testPushDownLocalSortAggWithSort(): Unit = {
+ // enable sort agg
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg")
+
+ checkResult(
+ """
+ |SELECT
+ | avg(deposit),
+ | sum(deposit),
+ | count(1),
+ | gender,
+ | age
+ |FROM
+ | AggregatableTable
+ |GROUP BY gender, age
+ |""".stripMargin,
+ Seq(
+ row(50, 50, 1, "f", 19),
+ row(200, 200, 1, "f", 20),
+ row(250, 750, 3, "m", 23),
+ row(126, 380, 3, "f", 25),
+ row(300, 300, 1, "m", 27),
+ row(170, 170, 1, "m", 28),
+ row(100, 100, 1, "m", 34))
+ )
+
+ // reset config
+ tEnv.getConfig.getConfiguration.setString(
+ ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "")
+ }
+
+ @Test
+ def testAggWithAuxGrouping(): Unit = {
Review comment:
removed this unsupported test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
}, {
"hash" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25622",
"triggerID" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aec0bc1492f607606f53ce208dc5c46aa07d2706 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25622)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
}, {
"hash" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ee07dc6ac1911d32636e5156c04820fb24d8b88 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615)
* aec0bc1492f607606f53ce208dc5c46aa07d2706 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa36f90cd26e229c5e0f7dfa66a756d4817007b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469)
* 70104a7e1bb8b322a8044143de0eb700a761794c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r737986628
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/AggregatePushDownSpec.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.source;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.expressions.AggregateExpression;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
+import org.apache.flink.table.planner.functions.aggfunctions.AvgAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.CountAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.Sum0AggFunction;
+import org.apache.flink.table.planner.plan.utils.AggregateInfo;
+import org.apache.flink.table.planner.plan.utils.AggregateInfoList;
+import org.apache.flink.table.planner.plan.utils.AggregateUtil;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import org.apache.calcite.rel.core.AggregateCall;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A sub-class of {@link SourceAbilitySpec} that can not only serialize/deserialize the aggregation
+ * to/from JSON, but also can push the local aggregate into a {@link SupportsAggregatePushDown}.
+ */
+@JsonTypeName("AggregatePushDown")
+public class AggregatePushDownSpec extends SourceAbilitySpecBase {
+
+ public static final String FIELD_NAME_INPUT_TYPE = "inputType";
+
+ public static final String FIELD_NAME_GROUPING_SETS = "groupingSets";
+
+ public static final String FIELD_NAME_AGGREGATE_CALLS = "aggregateCalls";
+
+ @JsonProperty(FIELD_NAME_INPUT_TYPE)
+ private final RowType inputType;
+
+ @JsonProperty(FIELD_NAME_GROUPING_SETS)
+ private final List<int[]> groupingSets;
+
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS)
+ private final List<AggregateCall> aggregateCalls;
+
+ @JsonCreator
+ public AggregatePushDownSpec(
+ @JsonProperty(FIELD_NAME_INPUT_TYPE) RowType inputType,
+ @JsonProperty(FIELD_NAME_GROUPING_SETS) List<int[]> groupingSets,
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS) List<AggregateCall> aggregateCalls,
+ @JsonProperty(FIELD_NAME_PRODUCED_TYPE) RowType producedType) {
+ super(producedType);
+
+ this.inputType = inputType;
+ this.groupingSets = new ArrayList<>(checkNotNull(groupingSets));
+ this.aggregateCalls = aggregateCalls;
+ }
+
+ @Override
+ public void apply(DynamicTableSource tableSource, SourceAbilityContext context) {
+ checkArgument(getProducedType().isPresent());
+ apply(
+ inputType,
+ groupingSets,
+ aggregateCalls,
+ getProducedType().get(),
+ tableSource,
+ context);
+ }
+
+ @Override
+ public String getDigests(SourceAbilityContext context) {
+ int[] grouping = groupingSets.get(0);
+ String groupingStr =
+ Arrays.stream(grouping)
+ .mapToObj(index -> inputType.getFieldNames().get(index))
+ .collect(Collectors.joining(","));
+
+ List<AggregateExpression> aggregateExpressions =
+ buildAggregateExpressions(inputType, aggregateCalls);
+ String aggFunctionsStr =
+ aggregateExpressions.stream()
+ .map(AggregateExpression::asSummaryString)
+ .collect(Collectors.joining(","));
+
+ return "aggregates=[grouping=["
+ + groupingStr
+ + "], aggFunctions=["
+ + aggFunctionsStr
+ + "]]";
+ }
+
+ public static boolean apply(
+ RowType inputType,
+ List<int[]> groupingSets,
+ List<AggregateCall> aggregateCalls,
+ RowType producedType,
+ DynamicTableSource tableSource,
+ SourceAbilityContext context) {
+ assert context.isBatchMode();
Review comment:
add `assert groupingSets.size() == 1`
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ pushLocalAggregateIntoScan(call, localAgg, oldScan, null);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan,
+ int[] calcRefFields) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets =
+ Collections.singletonList(
+ ArrayUtils.addAll(localAgg.grouping(), localAgg.auxGrouping()));
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+
+ // map arg index in aggregate to field index in scan through referred fields by calc.
+ if (calcRefFields != null) {
+ groupingSets = translateGroupingArgIndex(groupingSets, calcRefFields);
+ aggCallList = translateAggCallArgIndex(aggCallList, calcRefFields);
+ }
+
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType,
+ groupingSets,
+ aggCallList,
+ producedType,
+ newTableSource,
+ SourceAbilityContext.from(oldScan));
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ // create new source table with new spec and statistic.
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ Set<String> groupColumns =
+ Arrays.stream(groupingSets.get(0))
+ .boxed()
+ .map(idx -> inputType.getFieldNames().get(idx))
+ .collect(Collectors.toSet());
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable, groupColumns);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ localAgg.getRowType(),
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(newFlinkStatistic);
+
+ // transform to new nodes.
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(
+ TableSourceTable tableSourceTable, Set<String> groupColumns) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+
+ // Create new unique keys if there are group columns
+ Set<Set<String>> uniqueKeys = null;
+ if (!groupColumns.isEmpty()) {
+ uniqueKeys = new HashSet<>();
+ uniqueKeys.add(groupColumns);
+ }
+
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder()
+ .statistic(oldStatistic)
+ .uniqueKeys(uniqueKeys)
+ .tableStats(null)
+ .build();
+ }
+
+ protected boolean checkNoProjectionPushDown(BatchPhysicalTableSourceScan tableSourceScan) {
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ return tableSourceTable != null
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof ProjectPushDownSpec);
+ }
+
+ /**
+ * Currently, we only supports to push down aggregate above calc which has input ref only.
+ *
+ * @param calc BatchPhysicalCalc
+ * @return true if OK to be pushed down
+ */
+ protected boolean checkCalcInputRefOnly(BatchPhysicalCalc calc) {
Review comment:
isInputRefOnly ?
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.batch.sql.agg;
+
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.types.Row;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+
+/** Test for local aggregate push down. */
+public class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ public void before() {
+ super.before();
+ env().setParallelism(1); // set sink parallelism to 1
+
+ String testDataId = TestValuesTableFactory.registerData(TestData.personData());
+ String ddl =
+ "CREATE TABLE AggregatableTable (\n"
+ + " id int,\n"
+ + " age int,\n"
+ + " name string,\n"
+ + " height int,\n"
+ + " gender string,\n"
+ + " deposit bigint,\n"
+ + " points bigint,\n"
+ + " metadata_1 BIGINT METADATA,\n"
+ + " metadata_2 STRING METADATA\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'data-id' = '"
+ + testDataId
+ + "',\n"
+ + " 'filterable-fields' = 'id;age',\n"
+ + " 'readable-metadata' = 'metadata_1:BIGINT, metadata_2:STRING',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ tEnv().executeSql(ddl);
+
+ // partitioned table
+ String testDataId2 = TestValuesTableFactory.registerData(TestData.personData());
+ String ddl2 =
+ "CREATE TABLE AggregatableTable_Part (\n"
+ + " id int,\n"
+ + " age int,\n"
+ + " name string,\n"
+ + " height int,\n"
+ + " gender string,\n"
+ + " deposit bigint,\n"
+ + " points bigint,\n"
+ + " distance BIGINT,\n"
+ + " type STRING\n"
+ + ") PARTITIONED BY (type)\n"
+ + "WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'data-id' = '"
+ + testDataId
+ + "',\n"
+ + " 'filterable-fields' = 'id;age',\n"
+ + " 'partition-list' = 'type:A;type:B;type:C;type:D',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ tEnv().executeSql(ddl2);
+
+ // partitioned table
+ String testDataId3 = TestValuesTableFactory.registerData(TestData.personData());
Review comment:
ditto
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ pushLocalAggregateIntoScan(call, localAgg, oldScan, null);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan,
+ int[] calcRefFields) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets =
+ Collections.singletonList(
+ ArrayUtils.addAll(localAgg.grouping(), localAgg.auxGrouping()));
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+
+ // map arg index in aggregate to field index in scan through referred fields by calc.
+ if (calcRefFields != null) {
+ groupingSets = translateGroupingArgIndex(groupingSets, calcRefFields);
+ aggCallList = translateAggCallArgIndex(aggCallList, calcRefFields);
+ }
+
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType,
+ groupingSets,
+ aggCallList,
+ producedType,
+ newTableSource,
+ SourceAbilityContext.from(oldScan));
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ // create new source table with new spec and statistic.
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ Set<String> groupColumns =
+ Arrays.stream(groupingSets.get(0))
+ .boxed()
+ .map(idx -> inputType.getFieldNames().get(idx))
+ .collect(Collectors.toSet());
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable, groupColumns);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ localAgg.getRowType(),
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(newFlinkStatistic);
+
+ // transform to new nodes.
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(
+ TableSourceTable tableSourceTable, Set<String> groupColumns) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+
+ // Create new unique keys if there are group columns
+ Set<Set<String>> uniqueKeys = null;
+ if (!groupColumns.isEmpty()) {
+ uniqueKeys = new HashSet<>();
+ uniqueKeys.add(groupColumns);
+ }
+
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder()
+ .statistic(oldStatistic)
+ .uniqueKeys(uniqueKeys)
+ .tableStats(null)
+ .build();
+ }
+
+ protected boolean checkNoProjectionPushDown(BatchPhysicalTableSourceScan tableSourceScan) {
Review comment:
isProjectionNotPushDown ?
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -286,6 +298,9 @@ private static RowKind parseRowKind(String rowKindShortString) {
private static final ConfigOption<Integer> SINK_EXPECTED_MESSAGES_NUM =
ConfigOptions.key("sink-expected-messages-num").intType().defaultValue(-1);
+ private static final ConfigOption<Boolean> DISABLE_PROJECTION_PUSH_DOWN =
+ ConfigOptions.key("disable-projection-push-down").booleanType().defaultValue(false);
Review comment:
enable-projection-push-down ? which is more intuitive
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ pushLocalAggregateIntoScan(call, localAgg, oldScan, null);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan,
+ int[] calcRefFields) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets =
+ Collections.singletonList(
+ ArrayUtils.addAll(localAgg.grouping(), localAgg.auxGrouping()));
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+
+ // map arg index in aggregate to field index in scan through referred fields by calc.
+ if (calcRefFields != null) {
+ groupingSets = translateGroupingArgIndex(groupingSets, calcRefFields);
+ aggCallList = translateAggCallArgIndex(aggCallList, calcRefFields);
+ }
+
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType,
+ groupingSets,
+ aggCallList,
+ producedType,
+ newTableSource,
+ SourceAbilityContext.from(oldScan));
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ // create new source table with new spec and statistic.
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ Set<String> groupColumns =
+ Arrays.stream(groupingSets.get(0))
+ .boxed()
+ .map(idx -> inputType.getFieldNames().get(idx))
+ .collect(Collectors.toSet());
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable, groupColumns);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ localAgg.getRowType(),
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(newFlinkStatistic);
+
+ // transform to new nodes.
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(
+ TableSourceTable tableSourceTable, Set<String> groupColumns) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+
+ // Create new unique keys if there are group columns
+ Set<Set<String>> uniqueKeys = null;
+ if (!groupColumns.isEmpty()) {
+ uniqueKeys = new HashSet<>();
+ uniqueKeys.add(groupColumns);
+ }
+
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder()
+ .statistic(oldStatistic)
+ .uniqueKeys(uniqueKeys)
Review comment:
the `uniqueKeys` is not correct, because we just push down the local aggregate instead of global aggregate, the result between each parallelism may have overlap. So this method should return FlinkStatistic.UNKNOWN
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
Review comment:
canPushDown ?
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ pushLocalAggregateIntoScan(call, localAgg, oldScan, null);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan,
+ int[] calcRefFields) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets =
+ Collections.singletonList(
+ ArrayUtils.addAll(localAgg.grouping(), localAgg.auxGrouping()));
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+
+ // map arg index in aggregate to field index in scan through referred fields by calc.
+ if (calcRefFields != null) {
+ groupingSets = translateGroupingArgIndex(groupingSets, calcRefFields);
+ aggCallList = translateAggCallArgIndex(aggCallList, calcRefFields);
+ }
+
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType,
+ groupingSets,
+ aggCallList,
+ producedType,
+ newTableSource,
+ SourceAbilityContext.from(oldScan));
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ // create new source table with new spec and statistic.
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ Set<String> groupColumns =
+ Arrays.stream(groupingSets.get(0))
+ .boxed()
+ .map(idx -> inputType.getFieldNames().get(idx))
+ .collect(Collectors.toSet());
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable, groupColumns);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ localAgg.getRowType(),
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(newFlinkStatistic);
+
+ // transform to new nodes.
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(
+ TableSourceTable tableSourceTable, Set<String> groupColumns) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+
+ // Create new unique keys if there are group columns
+ Set<Set<String>> uniqueKeys = null;
+ if (!groupColumns.isEmpty()) {
+ uniqueKeys = new HashSet<>();
+ uniqueKeys.add(groupColumns);
+ }
+
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder()
+ .statistic(oldStatistic)
+ .uniqueKeys(uniqueKeys)
+ .tableStats(null)
+ .build();
+ }
+
+ protected boolean checkNoProjectionPushDown(BatchPhysicalTableSourceScan tableSourceScan) {
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ return tableSourceTable != null
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof ProjectPushDownSpec);
+ }
+
+ /**
+ * Currently, we only supports to push down aggregate above calc which has input ref only.
+ *
+ * @param calc BatchPhysicalCalc
+ * @return true if OK to be pushed down
+ */
+ protected boolean checkCalcInputRefOnly(BatchPhysicalCalc calc) {
+ RexProgram program = calc.getProgram();
+
+ // check if condition exists. All filters should have been pushed down.
+ if (program.getCondition() != null) {
+ return false;
+ }
+
+ return program.getExprList().stream().allMatch(RexInputRef.class::isInstance)
Review comment:
most case, we will convert RexLocalRef list to RexNode list via calc.getProgram()::expandLocalRef.
program.getProjectList().stream()
.map(calc.getProgram()::expandLocalRef)
.allMatch(RexInputRef.class::isInstance)
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalSortAggWithSortAndCalcIntoScanRule.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalSortAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalSort;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.utils.RexNodeExtractor;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local sort aggregate which with sort into a {@link
Review comment:
with sort and calc
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.batch.sql.agg;
+
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.types.Row;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+
+/** Test for local aggregate push down. */
+public class LocalAggregatePushDownITCase extends BatchTestBase {
+
+ @Before
+ public void before() {
+ super.before();
+ env().setParallelism(1); // set sink parallelism to 1
+
+ String testDataId = TestValuesTableFactory.registerData(TestData.personData());
+ String ddl =
+ "CREATE TABLE AggregatableTable (\n"
+ + " id int,\n"
+ + " age int,\n"
+ + " name string,\n"
+ + " height int,\n"
+ + " gender string,\n"
+ + " deposit bigint,\n"
+ + " points bigint,\n"
+ + " metadata_1 BIGINT METADATA,\n"
+ + " metadata_2 STRING METADATA\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'data-id' = '"
+ + testDataId
+ + "',\n"
+ + " 'filterable-fields' = 'id;age',\n"
+ + " 'readable-metadata' = 'metadata_1:BIGINT, metadata_2:STRING',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ tEnv().executeSql(ddl);
+
+ // partitioned table
+ String testDataId2 = TestValuesTableFactory.registerData(TestData.personData());
Review comment:
this test data is useless
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalSortAggWithCalcIntoScanRule.java
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalLocalSortAggregate;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.utils.RexNodeExtractor;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+
+/**
+ * Planner rule that tries to push a local sort aggregate which without sort into a {@link
+ * BatchPhysicalTableSourceScan} whose table is a {@link TableSourceTable} with a source supporting
+ * {@link SupportsAggregatePushDown}. The {@link
+ * OptimizerConfigOptions#TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED} need to be true.
+ *
+ * <p>Suppose we have the original physical plan:
+ *
+ * <pre>{@code
+ * BatchPhysicalSortAggregate (global)
+ * +- BatchPhysicalExchange (hash by group keys if group keys is not empty, else singleton)
+ * +- BatchPhysicalLocalSortAggregate (local)
+ * +- BatchPhysicalCalc (filed projection only)
+ * +- BatchPhysicalTableSourceScan
+ * }</pre>
+ *
+ * <p>This physical plan will be rewritten to:
+ *
+ * <pre>{@code
+ * BatchPhysicalSortAggregate (global)
+ * +- BatchPhysicalExchange (hash by group keys if group keys is not empty, else singleton)
+ * +- BatchPhysicalTableSourceScan (with local aggregate pushed down)
+ * }</pre>
+ */
+public class PushLocalSortAggWithCalcIntoScanRule extends PushLocalAggIntoScanRuleBase {
+ public static final PushLocalSortAggWithCalcIntoScanRule INSTANCE =
+ new PushLocalSortAggWithCalcIntoScanRule();
+
+ public PushLocalSortAggWithCalcIntoScanRule() {
+ super(
+ operand(
+ BatchPhysicalExchange.class,
+ operand(
+ BatchPhysicalLocalSortAggregate.class,
+ operand(
+ BatchPhysicalCalc.class,
+ operand(BatchPhysicalTableSourceScan.class, none())))),
+ "PushLocalSortAggWithCalcIntoScanRule");
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+ BatchPhysicalLocalSortAggregate localAggregate = call.rel(1);
+ BatchPhysicalCalc calc = call.rel(2);
+ BatchPhysicalTableSourceScan tableSourceScan = call.rel(3);
+
+ return checkCalcInputRefOnly(calc)
+ && checkNoProjectionPushDown(tableSourceScan)
+ && checkMatchesAggregatePushDown(call, localAggregate, tableSourceScan);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ BatchPhysicalLocalSortAggregate localHashAgg = call.rel(1);
+ BatchPhysicalCalc calc = call.rel(2);
+ BatchPhysicalTableSourceScan oldScan = call.rel(3);
+
+ int[] calcRefFields =
+ RexNodeExtractor.extractRefInputFields(calc.getProgram().getExprList());
Review comment:
program.getProjectList().stream()
.map(calc.getProgram()::expandLocalRef)
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/batch/sql/agg/LocalAggregatePushDownITCase.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.batch.sql.agg;
+
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.planner.runtime.utils.BatchTestBase;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.types.Row;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+
+/** Test for local aggregate push down. */
+public class LocalAggregatePushDownITCase extends BatchTestBase {
Review comment:
please add a test case about push local aggregate with auxGrouping
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.ProjectPushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilityContext;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalCalc;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexProgram;
+import org.apache.calcite.rex.RexSlot;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * whose table is a {@link TableSourceTable} with a source supporting {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean checkMatchesAggregatePushDown(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ pushLocalAggregateIntoScan(call, localAgg, oldScan, null);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan,
+ int[] calcRefFields) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets =
+ Collections.singletonList(
+ ArrayUtils.addAll(localAgg.grouping(), localAgg.auxGrouping()));
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+
+ // map arg index in aggregate to field index in scan through referred fields by calc.
+ if (calcRefFields != null) {
+ groupingSets = translateGroupingArgIndex(groupingSets, calcRefFields);
+ aggCallList = translateAggCallArgIndex(aggCallList, calcRefFields);
+ }
+
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType,
+ groupingSets,
+ aggCallList,
+ producedType,
+ newTableSource,
+ SourceAbilityContext.from(oldScan));
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ // create new source table with new spec and statistic.
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ Set<String> groupColumns =
+ Arrays.stream(groupingSets.get(0))
+ .boxed()
+ .map(idx -> inputType.getFieldNames().get(idx))
+ .collect(Collectors.toSet());
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable, groupColumns);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ localAgg.getRowType(),
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(newFlinkStatistic);
+
+ // transform to new nodes.
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(
+ TableSourceTable tableSourceTable, Set<String> groupColumns) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+
+ // Create new unique keys if there are group columns
+ Set<Set<String>> uniqueKeys = null;
+ if (!groupColumns.isEmpty()) {
+ uniqueKeys = new HashSet<>();
+ uniqueKeys.add(groupColumns);
+ }
+
+ // Remove tableStats after all aggregates have been pushed down
+ return FlinkStatistic.builder()
+ .statistic(oldStatistic)
+ .uniqueKeys(uniqueKeys)
+ .tableStats(null)
+ .build();
+ }
+
+ protected boolean checkNoProjectionPushDown(BatchPhysicalTableSourceScan tableSourceScan) {
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ return tableSourceTable != null
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof ProjectPushDownSpec);
+ }
+
+ /**
+ * Currently, we only supports to push down aggregate above calc which has input ref only.
+ *
+ * @param calc BatchPhysicalCalc
+ * @return true if OK to be pushed down
+ */
+ protected boolean checkCalcInputRefOnly(BatchPhysicalCalc calc) {
+ RexProgram program = calc.getProgram();
+
+ // check if condition exists. All filters should have been pushed down.
+ if (program.getCondition() != null) {
+ return false;
+ }
+
+ return program.getExprList().stream().allMatch(RexInputRef.class::isInstance)
+ && !program.getProjectList().isEmpty();
+ }
+
+ protected int[] getRefFiledIndexFromCalc(BatchPhysicalCalc calc) {
Review comment:
use RexNodeExtractor.extractRefInputField instead
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for rules that extend {@link PushLocalAggIntoScanRuleBase} to push down local aggregates
+ * into table source.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
+ protected BatchTableTestUtil util = batchTestUtil(new TableConfig());
+
+ @Before
+ public void setup() {
+ TableConfig tableConfig = util.tableEnv().getConfig();
+ tableConfig
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ String ddl =
+ "CREATE TABLE inventory (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id;type',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl);
+
+ String ddl2 =
+ "CREATE TABLE inventory_meta (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING,\n"
+ + " metadata_1 BIGINT METADATA,\n"
+ + " metadata_2 STRING METADATA,\n"
+ + " PRIMARY KEY (`id`) NOT ENFORCED\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id;type',\n"
+ + " 'readable-metadata' = 'metadata_1:BIGINT, metadata_2:STRING',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl2);
+
+ // partitioned table
+ String ddl3 =
+ "CREATE TABLE inventory_part (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") PARTITIONED BY (type)\n"
+ + "WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id;type',\n"
+ + " 'partition-list' = 'type:a;type:b',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl3);
+
+ // disable projection push down
+ String ddl4 =
+ "CREATE TABLE inventory_no_proj (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ")\n"
+ + "WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id;type',\n"
+ + " 'disable-projection-push-down' = 'true',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl4);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testDisablePushDownLocalAgg() {
+ // disable push down local agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false);
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithoutGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithoutSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggAfterFilterPushDown() {
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " where id = 123\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithMetadata() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " max(metadata_1),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory_meta\n"
+ + " where id = 123\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithPartition() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " type,\n"
+ + " name\n"
+ + "FROM inventory_part\n"
+ + " where type in ('a', 'b') and id = 123\n"
+ + " group by type, name");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithoutProjectionPushDown() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory_no_proj\n"
+ + " where id = 123\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testCannotPushDownLocalAggWithAuxGrouping() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " id, name, count(*)\n"
+ + "FROM inventory_meta\n"
+ + " group by id, name, abs(amount)");
Review comment:
remove abs(amount), otherwise the local aggregate with auxGrouping can not be pushed down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ee07dc6ac1911d32636e5156c04820fb24d8b88 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r738899089
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala
##########
@@ -1002,6 +1001,12 @@ abstract class TableTestUtil(
.getConfiguration
.set(ExecutionOptions.BATCH_SHUFFLE_MODE, BatchShuffleMode.ALL_EXCHANGES_PIPELINED)
+ // Disable push down aggregates to avoid conflicts with existing test cases that verify plans.
+ tableEnv.getConfig
+ .getConfiguration
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, false)
Review comment:
pushed a new commit https://github.com/apache/flink/pull/17344/commits/2ee07dc6ac1911d32636e5156c04820fb24d8b88 to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735221321
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
+ protected BatchTableTestUtil util = batchTestUtil(new TableConfig());
+
+ @Before
+ public void setup() {
+ TableConfig tableConfig = util.tableEnv().getConfig();
+ tableConfig
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ String ddl =
+ "CREATE TABLE inventory (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testDisablePushDownLocalAgg() {
+ // disable push down local agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false);
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithoutGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithoutSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithAuxGrouping() {
Review comment:
updated this test case. It is a local aggregate with `auxGrouping` now.
However, there is a `Calc` for a column in grouping. So this is not supported to be pushed down.
If we compose a query without `Calc`, the whole local aggregate will be removed automatically since `PRIMARY KEY`, even without our new rules.
So we take `local aggregate` with `auxGrouping` as unsupported case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735225200
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
+ protected BatchTableTestUtil util = batchTestUtil(new TableConfig());
+
+ @Before
+ public void setup() {
+ TableConfig tableConfig = util.tableEnv().getConfig();
+ tableConfig
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ String ddl =
+ "CREATE TABLE inventory (\n"
+ + " id BIGINT,\n"
+ + " name STRING,\n"
+ + " amount BIGINT,\n"
+ + " price BIGINT,\n"
+ + " type STRING\n"
+ + ") WITH (\n"
+ + " 'connector' = 'values',\n"
+ + " 'filterable-fields' = 'id',\n"
+ + " 'bounded' = 'true'\n"
+ + ")";
+ util.tableEnv().executeSql(ddl);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+ }
+
+ @Test
+ public void testDisablePushDownLocalAgg() {
+ // disable push down local agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ false);
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED,
+ true);
+ }
+
+ @Test
+ public void testCanPushDownLocalHashAggWithoutGroup() {
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithoutSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " min(id),\n"
+ + " max(amount),\n"
+ + " sum(price),\n"
+ + " avg(price),\n"
+ + " count(id)\n"
+ + "FROM inventory");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalSortAggWithSort() {
+ // enable sort agg
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "HashAgg");
+
+ util.verifyRelPlan(
+ "SELECT\n"
+ + " sum(amount),\n"
+ + " name,\n"
+ + " type\n"
+ + "FROM inventory\n"
+ + " group by name, type");
+
+ // reset config
+ util.getTableEnv()
+ .getConfig()
+ .getConfiguration()
+ .setString(ExecutionConfigOptions.TABLE_EXEC_DISABLED_OPERATORS, "");
+ }
+
+ @Test
+ public void testCanPushDownLocalAggWithAuxGrouping() {
Review comment:
In short, we can't set a `local aggregate` with `auxGrouping` but without `Calc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735219348
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##########
@@ -867,27 +885,118 @@ public String asSummaryString() {
allPartitions.isEmpty()
? Collections.singletonList(Collections.emptyMap())
: allPartitions;
+
int numRetained = 0;
for (Map<String, String> partition : keys) {
- for (Row row : data.get(partition)) {
+ Collection<Row> rowsInPartition = data.get(partition);
+
+ // handle predicates and projection
+ List<Row> rowsRetained =
+ rowsInPartition.stream()
+ .filter(
+ row ->
+ FilterUtils.isRetainedAfterApplyingFilterPredicates(
+ filterPredicates, getValueGetter(row)))
+ .map(
+ row -> {
+ Row projectedRow = projectRow(row);
+ projectedRow.setKind(row.getKind());
+ return projectedRow;
+ })
+ .collect(Collectors.toList());
+
+ // handle aggregates
+ if (!aggregateExpressions.isEmpty()) {
+ rowsRetained = applyAggregatesToRows(rowsRetained);
+ }
+
+ // handle row data
+ for (Row row : rowsRetained) {
+ final RowData rowData = (RowData) converter.toInternal(row);
+ if (rowData != null) {
+ if (numRetained >= numElementToSkip) {
Review comment:
skipping original elements now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e795eef112edd7a5f705596151cacee93d53c03 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403)
* 0a76c10b817de7e5eaa8ec5a790ffdbf2e929252 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472)
* 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-929774236
@wuchong @godfreyhe Please take a look, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-929774236
@wuchong @godfreyhe Please take a look, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a76c10b817de7e5eaa8ec5a790ffdbf2e929252 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585)
* 2ee07dc6ac1911d32636e5156c04820fb24d8b88 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
}, {
"hash" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25622",
"triggerID" : "aec0bc1492f607606f53ce208dc5c46aa07d2706",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ee07dc6ac1911d32636e5156c04820fb24d8b88 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615)
* aec0bc1492f607606f53ce208dc5c46aa07d2706 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25622)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r740240008
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsAggregatePushDown.java
##########
@@ -124,8 +124,6 @@
*
* <p>Regardless if this interface is implemented or not, a final aggregation is always applied in a
* subsequent operation after the source.
- *
- * <p>Note: currently, the {@link SupportsAggregatePushDown} is not supported by planner.
Review comment:
Hi @twalthr I pushed this for doc improvement: https://github.com/apache/flink/pull/17630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r715005516
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/OptimizerConfigOptions.java
##########
@@ -93,6 +93,16 @@
+ TABLE_OPTIMIZER_REUSE_SUB_PLAN_ENABLED.key()
+ " is true.");
+ @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH)
+ public static final ConfigOption<Boolean> TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED =
+ key("table.optimizer.source.aggregate-pushdown-enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "When it is true, the optimizer will push down the local aggregates into "
+ + "the TableSource which implements SupportsAggregatePushDown. "
+ + "Default value is false.");
Review comment:
The default doesn't need to be stated here. It is recorded above and the docs generator will use that, so this leads to duplication.
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/OptimizerConfigOptions.java
##########
@@ -93,6 +93,16 @@
+ TABLE_OPTIMIZER_REUSE_SUB_PLAN_ENABLED.key()
+ " is true.");
+ @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH)
+ public static final ConfigOption<Boolean> TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED =
+ key("table.optimizer.source.aggregate-pushdown-enabled")
+ .booleanType()
+ .defaultValue(false)
Review comment:
Is there a reason to disable this by default? I would expect that a source implements the interface only if it does so properly. By disabling this we hide this optimization from the majority of users.
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * which table is a {@link TableSourceTable}. And the table source in the table is a {@link
Review comment:
```suggestion
* whose table is a {@link TableSourceTable} with a source supporting {@link
```
Same applies to subclasses
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoScanRuleBase.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.plan.abilities.source.AggregatePushDownSpec;
+import org.apache.flink.table.planner.plan.abilities.source.SourceAbilitySpec;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalExchange;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalGroupAggregateBase;
+import org.apache.flink.table.planner.plan.nodes.physical.batch.BatchPhysicalTableSourceScan;
+import org.apache.flink.table.planner.plan.schema.TableSourceTable;
+import org.apache.flink.table.planner.plan.stats.FlinkStatistic;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.ShortcutUtils;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.AggregateCall;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Planner rule that tries to push a local aggregator into an {@link BatchPhysicalTableSourceScan}
+ * which table is a {@link TableSourceTable}. And the table source in the table is a {@link
+ * SupportsAggregatePushDown}.
+ *
+ * <p>The aggregate push down does not support a number of more complex statements at present:
+ *
+ * <ul>
+ * <li>complex grouping operations such as ROLLUP, CUBE, or GROUPING SETS.
+ * <li>expressions inside the aggregation function call: such as sum(a * b).
+ * <li>aggregations with ordering.
+ * <li>aggregations with filter.
+ * </ul>
+ */
+public abstract class PushLocalAggIntoScanRuleBase extends RelOptRule {
+
+ public PushLocalAggIntoScanRuleBase(RelOptRuleOperand operand, String description) {
+ super(operand, description);
+ }
+
+ protected boolean isMatch(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase aggregate,
+ BatchPhysicalTableSourceScan tableSourceScan) {
+ TableConfig tableConfig = ShortcutUtils.unwrapContext(call.getPlanner()).getTableConfig();
+ if (!tableConfig
+ .getConfiguration()
+ .getBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED)) {
+ return false;
+ }
+
+ if (aggregate.isFinal() || aggregate.getAggCallList().isEmpty()) {
+ return false;
+ }
+ List<AggregateCall> aggCallList =
+ JavaScalaConversionUtil.toJava(aggregate.getAggCallList());
+ for (AggregateCall aggCall : aggCallList) {
+ if (aggCall.isDistinct()
+ || aggCall.isApproximate()
+ || aggCall.getArgList().size() > 1
+ || aggCall.hasFilter()
+ || !aggCall.getCollation().getFieldCollations().isEmpty()) {
+ return false;
+ }
+ }
+ TableSourceTable tableSourceTable = tableSourceScan.tableSourceTable();
+ // we can not push aggregates twice
+ return tableSourceTable != null
+ && tableSourceTable.tableSource() instanceof SupportsAggregatePushDown
+ && Arrays.stream(tableSourceTable.abilitySpecs())
+ .noneMatch(spec -> spec instanceof AggregatePushDownSpec);
+ }
+
+ protected void pushLocalAggregateIntoScan(
+ RelOptRuleCall call,
+ BatchPhysicalGroupAggregateBase localAgg,
+ BatchPhysicalTableSourceScan oldScan) {
+ RowType inputType = FlinkTypeFactory.toLogicalRowType(oldScan.getRowType());
+ List<int[]> groupingSets = Arrays.asList(localAgg.grouping(), localAgg.auxGrouping());
+ List<AggregateCall> aggCallList = JavaScalaConversionUtil.toJava(localAgg.getAggCallList());
+ RowType producedType = FlinkTypeFactory.toLogicalRowType(localAgg.getRowType());
+
+ TableSourceTable oldTableSourceTable = oldScan.tableSourceTable();
+ DynamicTableSource newTableSource = oldScan.tableSource().copy();
+
+ boolean isPushDownSuccess =
+ AggregatePushDownSpec.apply(
+ inputType, groupingSets, aggCallList, producedType, newTableSource);
+
+ if (!isPushDownSuccess) {
+ // aggregate push down failed, just return without changing any nodes.
+ return;
+ }
+
+ FlinkStatistic newFlinkStatistic = getNewFlinkStatistic(oldTableSourceTable);
+ AggregatePushDownSpec aggregatePushDownSpec =
+ new AggregatePushDownSpec(inputType, groupingSets, aggCallList, producedType);
+
+ TableSourceTable newTableSourceTable =
+ oldTableSourceTable
+ .copy(
+ newTableSource,
+ newFlinkStatistic,
+ new SourceAbilitySpec[] {aggregatePushDownSpec})
+ .copy(localAgg.getRowType());
+ BatchPhysicalTableSourceScan newScan =
+ oldScan.copy(oldScan.getTraitSet(), newTableSourceTable);
+ BatchPhysicalExchange oldExchange = call.rel(0);
+ BatchPhysicalExchange newExchange =
+ oldExchange.copy(oldExchange.getTraitSet(), newScan, oldExchange.getDistribution());
+ call.transformTo(newExchange);
+ }
+
+ private FlinkStatistic getNewFlinkStatistic(TableSourceTable tableSourceTable) {
+ FlinkStatistic oldStatistic = tableSourceTable.getStatistic();
+ FlinkStatistic newStatistic;
Review comment:
nit: this variable is unnecessary, you can return directly within if/else.
##########
File path: flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/AggregatePushDownSpec.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.abilities.source;
+
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown;
+import org.apache.flink.table.expressions.AggregateExpression;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
+import org.apache.flink.table.planner.functions.aggfunctions.AvgAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.CountAggFunction;
+import org.apache.flink.table.planner.functions.aggfunctions.Sum0AggFunction;
+import org.apache.flink.table.planner.plan.utils.AggregateInfo;
+import org.apache.flink.table.planner.plan.utils.AggregateInfoList;
+import org.apache.flink.table.planner.plan.utils.AggregateUtil;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A sub-class of {@link SourceAbilitySpec} that can not only serialize/deserialize the aggregation
+ * to/from JSON, but also can push the local aggregate into a {@link SupportsAggregatePushDown}.
+ */
+@JsonTypeName("AggregatePushDown")
+public class AggregatePushDownSpec extends SourceAbilitySpecBase {
+
+ public static final String FIELD_NAME_INPUT_TYPE = "inputType";
+
+ public static final String FIELD_NAME_GROUPING_SETS = "groupingSets";
+
+ public static final String FIELD_NAME_AGGREGATE_CALLS = "aggregateCalls";
+
+ @JsonProperty(FIELD_NAME_INPUT_TYPE)
+ private final RowType inputType;
+
+ @JsonProperty(FIELD_NAME_GROUPING_SETS)
+ private final List<int[]> groupingSets;
+
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS)
+ private final List<AggregateCall> aggregateCalls;
+
+ @JsonCreator
+ public AggregatePushDownSpec(
+ @JsonProperty(FIELD_NAME_INPUT_TYPE) RowType inputType,
+ @JsonProperty(FIELD_NAME_GROUPING_SETS) List<int[]> groupingSets,
+ @JsonProperty(FIELD_NAME_AGGREGATE_CALLS) List<AggregateCall> aggregateCalls,
+ @JsonProperty(FIELD_NAME_PRODUCED_TYPE) RowType producedType) {
+ super(producedType);
+
+ this.inputType = inputType;
+ this.groupingSets = new ArrayList<>(checkNotNull(groupingSets));
+ this.aggregateCalls = aggregateCalls;
+ }
+
+ @Override
+ public void apply(DynamicTableSource tableSource, SourceAbilityContext context) {
+ checkArgument(getProducedType().isPresent());
+ apply(inputType, groupingSets, aggregateCalls, getProducedType().get(), tableSource);
+ }
+
+ @Override
+ public String getDigests(SourceAbilityContext context) {
+ String extraDigest;
+ String groupingStr = "";
+ int[] grouping = ArrayUtils.addAll(groupingSets.get(0), groupingSets.get(1));
+ if (grouping.length > 0) {
Review comment:
Here and below, the `if` are unnecessary. Iterating over empty lists works just fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r715300511
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/OptimizerConfigOptions.java
##########
@@ -93,6 +93,16 @@
+ TABLE_OPTIMIZER_REUSE_SUB_PLAN_ENABLED.key()
+ " is true.");
+ @Documentation.TableOption(execMode = Documentation.ExecMode.BATCH)
+ public static final ConfigOption<Boolean> TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED =
+ key("table.optimizer.source.aggregate-pushdown-enabled")
+ .booleanType()
+ .defaultValue(false)
Review comment:
Actually it can be `true` by default, just like `table.optimizer.source.predicate-pushdown-enabled`. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 479edc317b5315d56bdc0a3cdc06dcb3f97bcda6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454)
* 0fa36f90cd26e229c5e0f7dfa66a756d4817007b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735219201
##########
File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -131,4 +130,23 @@ class TableSourceTable(
flinkContext,
abilitySpecs ++ newAbilitySpecs)
}
+
+ /**
+ * Creates a copy of this table, changing the rowType
+ *
+ * @param newRowType new row type
+ * @return New TableSourceTable instance with new row type
+ */
+ def copy(newRowType: RelDataType): TableSourceTable = {
Review comment:
changed this `copy` with `FlinkStatistic`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r735222457
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/rules/physical/batch/PushLocalAggIntoTableSourceScanRuleTest.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.batch;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.planner.functions.aggfunctions.CollectAggFunction;
+import org.apache.flink.table.planner.utils.BatchTableTestUtil;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test for {@link PushLocalHashAggIntoScanRule}, {@link PushLocalSortAggWithSortIntoScanRule} and
+ * {@link PushLocalSortAggWithoutSortIntoScanRule}.
+ */
+public class PushLocalAggIntoTableSourceScanRuleTest extends TableTestBase {
Review comment:
added above test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e795eef112edd7a5f705596151cacee93d53c03 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r740148228
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsAggregatePushDown.java
##########
@@ -124,8 +124,6 @@
*
* <p>Regardless if this interface is implemented or not, a final aggregation is always applied in a
* subsequent operation after the source.
- *
- * <p>Note: currently, the {@link SupportsAggregatePushDown} is not supported by planner.
Review comment:
Hi @twalthr, the strategy is still `all or nothing` here. `applyAggregates` returns a bool value, meaning whether all local aggregates are accepted by underlying data source or not at all.
The docs could be improved, like mentioning the new option, etc.
I could push a hotfix for docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a76c10b817de7e5eaa8ec5a790ffdbf2e929252 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe closed pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
godfreyhe closed pull request #17344:
URL: https://github.com/apache/flink/pull/17344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e795eef112edd7a5f705596151cacee93d53c03 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403)
* 0a76c10b817de7e5eaa8ec5a790ffdbf2e929252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17344:
URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24454",
"triggerID" : "479edc317b5315d56bdc0a3cdc06dcb3f97bcda6",
"triggerType" : "PUSH"
}, {
"hash" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24469",
"triggerID" : "0fa36f90cd26e229c5e0f7dfa66a756d4817007b",
"triggerType" : "PUSH"
}, {
"hash" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472",
"triggerID" : "70104a7e1bb8b322a8044143de0eb700a761794c",
"triggerType" : "PUSH"
}, {
"hash" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493",
"triggerID" : "4ad29008075dc93463c07ce098cd7b1086104d7b",
"triggerType" : "PUSH"
}, {
"hash" : "4e795eef112edd7a5f705596151cacee93d53c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25403",
"triggerID" : "4e795eef112edd7a5f705596151cacee93d53c03",
"triggerType" : "PUSH"
}, {
"hash" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585",
"triggerID" : "0a76c10b817de7e5eaa8ec5a790ffdbf2e929252",
"triggerType" : "PUSH"
}, {
"hash" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615",
"triggerID" : "2ee07dc6ac1911d32636e5156c04820fb24d8b88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a76c10b817de7e5eaa8ec5a790ffdbf2e929252 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25585)
* 2ee07dc6ac1911d32636e5156c04820fb24d8b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25615)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] iyupeng commented on a change in pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
Posted by GitBox <gi...@apache.org>.
iyupeng commented on a change in pull request #17344:
URL: https://github.com/apache/flink/pull/17344#discussion_r738893985
##########
File path: flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala
##########
@@ -1002,6 +1001,12 @@ abstract class TableTestUtil(
.getConfiguration
.set(ExecutionOptions.BATCH_SHUFFLE_MODE, BatchShuffleMode.ALL_EXCHANGES_PIPELINED)
+ // Disable push down aggregates to avoid conflicts with existing test cases that verify plans.
+ tableEnv.getConfig
+ .getConfiguration
+ .setBoolean(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_SOURCE_AGGREGATE_PUSHDOWN_ENABLED, false)
Review comment:
Hello @godfreyhe, thanks for the comments.
2 existing test cases with `local aggregate` and `'connector' = 'values'` can be affected if we enable `aggregate push down`:
1. RankTest.scala#L172, `testCreateViewWithRowNumber`
2. TableSourceTest.scala#L118, `testProjectWithoutInputRef`
We need to change the plans of these 2 test cases in `.xml` files if we remove above configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org