You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/08 17:11:33 UTC
[GitHub] [flink] wuchong opened a new pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
wuchong opened a new pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Travis CI to do that following [this guide](https://flink.apache.org/contributing/contribute-code.html#open-a-pull-request).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Current retraction machanism to support more message kinds (insert/delete/update_before/update_after). And make the new machanism more extensible for future features and more readable.
## Brief change log
- 1st commit: introduce `FlinkChangelogModeInferenceProgram` with 2 traits and 2 satisfy visitor to infer changelog mode for every nodes.
- 2nd commit: Remove interfaces and traits and rules of legacy retraction mechanism entirely
- 3rd commit: Use new changelog trait description in plan tests and update all plan xml files
- 4th commit: Fix remaining tests because of the validation change and plan change
## Verifying this change
This pull request is verified with existing tests.
But some tests are modified to have the new changelog trait description instead of the legacy AccMode trait.
The plan change in `DagOptimizationTest` is reasonable, because we can reuse more nodes after refactoring.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* b8ea858f10d83717561e894c71e5b0b49fbd85f4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160141637) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417)
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407318612
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
Review comment:
We can't move it into `visitRankStrategies`, because `rank.copy(RankProcessStrategy)` is not a common interface of `StreamExecRank` and `StreamExecSortLimit`. I'm not sure whether we should introduce a common abstract for `StreamExecRank` and `StreamExecSortLimit`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407438419
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/ModifyKindSet.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * The set of modify operations contained in a changelog.
+ *
+ * @see ModifyKind
+ */
+public class ModifyKindSet {
+
+ /**
+ * Insert-only modify kind set.
+ */
+ public static final ModifyKindSet INSERT_ONLY = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .build();
+
+ /**
+ * A modify kind set contains all change operations.
+ */
+ public static final ModifyKindSet ALL_CHANGES = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ .addContainedKind(ModifyKind.DELETE)
+ .build();
+
+ private final Set<ModifyKind> kinds;
+
+ private ModifyKindSet(Set<ModifyKind> kinds) {
+ this.kinds = Collections.unmodifiableSet(kinds);
+ }
+
+ public Set<ModifyKind> getContainedKinds() {
+ return kinds;
+ }
+
+ public boolean contains(ModifyKind kind) {
+ return kinds.contains(kind);
+ }
+
+ public boolean containsOnly(ModifyKind kind) {
+ return kinds.size() == 1 && kinds.contains(kind);
+ }
+
+ public boolean isInsertOnly() {
+ return containsOnly(ModifyKind.INSERT);
+ }
+
+ public int size() {
+ return kinds.size();
+ }
+
+ public boolean isEmpty() {
+ return kinds.isEmpty();
+ }
+
+ /**
+ * Returns a new set of ModifyKind which is the difference between two sets.
+ * It is also equal to {@code this.kinds - that.kinds}. For example:
+ * [I,U,D] diff [I] = [U,D]
+ * [I,U] diff [U,D] = [I]
+ * [I,U,D] diff [I,U,D] = []
+ */
+ public ModifyKindSet diff(ModifyKindSet other) {
+ Set<ModifyKind> result = EnumSet.noneOf(ModifyKind.class);
+ result.addAll(this.kinds);
+ result.removeAll(other.kinds);
+ return new ModifyKindSet(result);
+ }
+
+ /**
+ * Returns a new ModifyKindSet with all kinds set in both this set and in another set.
+ */
+ public ModifyKindSet intersect(ModifyKindSet other) {
+ Builder builder = new Builder();
+ for (ModifyKind kind : other.getContainedKinds()) {
+ if (this.contains(kind)) {
+ builder.addContainedKind(kind);
+ }
+ }
+ return builder.build();
+ }
+
+ /**
+ * Returns a new ModifyKindSet with the union of the other ModifyKindSet.
+ */
+ public ModifyKindSet union(ModifyKindSet other) {
+ return union(this, other);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ModifyKindSet that = (ModifyKindSet) o;
+ return Objects.equals(kinds, that.kinds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(kinds);
+ }
+
+ @Override
+ public String toString() {
+ if (kinds.isEmpty()) {
Review comment:
This is on purpose. Sink will produce emtpy `ModifyKindSet`, because it output nothing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-613507721
Thanks for the reviewing. I have updated the branch to not satisfy ONLY_UPDATE_BEFORE for filters.
And added tests in `SinkTest` and `TableSinkITCase`. Which should already fix FLINK-9528.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407320577
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/UpdateKindTrait.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+/**
+ * UpdateKindTrait is used to describe the kind of update operation.
+ */
+class UpdateKindTrait(val updateKind: UpdateKind) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: UpdateKindTrait =>
+ // should totally match
+ other.updateKind == this.updateKind
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = UpdateKindTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = updateKind.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: UpdateKindTrait => this.updateKind.equals(t.updateKind)
+ case _ => false
+ }
+
+ override def toString: String = s"[${updateKind.toString}]"
+}
+
+object UpdateKindTrait {
+ val NO_UPDATE = new UpdateKindTrait(UpdateKind.NO_UPDATE)
Review comment:
I will add some comments here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-614040957
Merging...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407966473
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/api/stream/ExplainTest.xml
##########
@@ -45,11 +45,11 @@ LogicalProject(EXPR$0=[$1])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable1]])
== Optimized Logical Plan ==
-Calc(select=[EXPR$0], updateAsRetraction=[false], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
-+- GroupAggregate(groupBy=[a], select=[a, COUNT(*) AS EXPR$0], updateAsRetraction=[false], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- Exchange(distribution=[hash[a]], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- Calc(select=[a], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- DataStreamScan(table=[[default_catalog, default_database, MyTable1]], fields=[a, b, c], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
+Calc(select=[EXPR$0], changelogMode=[I,UA]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
Review comment:
why this operator is ignoring update before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611081869
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 (Wed Apr 08 17:14:03 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 6d34187c1dde98f1fc8573063e298bd481224688 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160017412) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407332545
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.xml
##########
@@ -313,17 +313,17 @@ LogicalProject(a1=[$1], b1=[$3])
</Resource>
<Resource name="planAfter">
<![CDATA[
-Join(joinType=[InnerJoin], where=[=(a1, b1)], select=[a1, b1], leftInputSpec=[JoinKeyContainsUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey], updateAsRetraction=[false], accMode=[Acc])
-:- Exchange(distribution=[hash[a1]], updateAsRetraction=[false], accMode=[Acc])
-: +- GroupAggregate(groupBy=[a1], select=[a1], updateAsRetraction=[false], accMode=[Acc])
-: +- Exchange(distribution=[hash[a1]], updateAsRetraction=[true], accMode=[Acc])
-: +- Calc(select=[a1, a2], updateAsRetraction=[true], accMode=[Acc])
-: +- TableSourceScan(table=[[default_catalog, default_database, A, source: [TestTableSource(a1, a2, a3)]]], fields=[a1, a2, a3], updateAsRetraction=[true], accMode=[Acc])
-+- Exchange(distribution=[hash[b1]], updateAsRetraction=[false], accMode=[Acc])
- +- GroupAggregate(groupBy=[b1], select=[b1], updateAsRetraction=[false], accMode=[Acc])
- +- Exchange(distribution=[hash[b1]], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[b1, b2], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, B, source: [TestTableSource(b1, b2, b3)]]], fields=[b1, b2, b3], updateAsRetraction=[true], accMode=[Acc])
+Join(joinType=[InnerJoin], where=[=(a1, b1)], select=[a1, b1], leftInputSpec=[JoinKeyContainsUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey], changelogMode=[I,UA,D])
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407320537
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
Review comment:
I added a `applyRankStrategy: RankProcessStrategy => StreamPhysicalRel` paramter to the method `visitRankStrategies` to make it possible move rank.copy into this method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407135526
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTraitDef.scala
##########
@@ -21,61 +21,30 @@ package org.apache.flink.table.planner.plan.`trait`
import org.apache.calcite.plan.{RelOptPlanner, RelTraitDef}
import org.apache.calcite.rel.RelNode
-/**
- * Definition of the [[UpdateAsRetractionTrait]].
- */
-class UpdateAsRetractionTraitDef extends RelTraitDef[UpdateAsRetractionTrait] {
- override def convert(
- planner: RelOptPlanner,
- rel: RelNode,
- toTrait: UpdateAsRetractionTrait,
- allowInfiniteCostConverters: Boolean): RelNode = {
+class ModifyKindSetTraitDef extends RelTraitDef[ModifyKindSetTrait] {
- rel.copy(rel.getTraitSet.plus(toTrait), rel.getInputs)
- }
-
- override def canConvert(
- planner: RelOptPlanner,
- fromTrait: UpdateAsRetractionTrait,
- toTrait: UpdateAsRetractionTrait): Boolean = true
-
- override def getTraitClass: Class[UpdateAsRetractionTrait] = classOf[UpdateAsRetractionTrait]
+ override def getTraitClass: Class[ModifyKindSetTrait] = classOf[ModifyKindSetTrait]
override def getSimpleName: String = this.getClass.getSimpleName
- override def getDefault: UpdateAsRetractionTrait = UpdateAsRetractionTrait.DEFAULT
-}
-
-object UpdateAsRetractionTraitDef {
- val INSTANCE = new UpdateAsRetractionTraitDef
-}
-
-/**
- * Definition of the [[AccModeTrait]].
- */
-class AccModeTraitDef extends RelTraitDef[AccModeTrait] {
-
override def convert(
planner: RelOptPlanner,
rel: RelNode,
- toTrait: AccModeTrait,
+ toTrait: ModifyKindSetTrait,
allowInfiniteCostConverters: Boolean): RelNode = {
-
rel.copy(rel.getTraitSet.plus(toTrait), rel.getInputs)
}
override def canConvert(
planner: RelOptPlanner,
- fromTrait: AccModeTrait,
- toTrait: AccModeTrait): Boolean = true
+ fromTrait: ModifyKindSetTrait,
+ toTrait: ModifyKindSetTrait): Boolean = true
Review comment:
we should throw exception here which means `ModifyKindSetTraitDef` does not support volcano planner now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r408223187
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,626 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNone, onlyAfterOrNone}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical node.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NONE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node's behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ createNewNode(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ createNewNode(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ createNewNode(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ createNewNode(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ createNewNode(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank and SortLimit supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ createNewNode(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ createNewNode(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ createNewNode(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ createNewNode(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (innerOrSemi) {
+ // forward left and right modify operations
+ new ModifyKindSetTrait(leftKindSet.union(rightKindSet))
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ createNewNode(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ createNewNode(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ createNewNode(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ createNewNode(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ createNewNode(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ createNewNode(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def createNewNode(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.minus(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfies required traits by input nodes of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val onlyAfter = onlyAfterOrNone(childModifyKindSet)
+ val beforeAndAfter = beforeAfterOrNone(childModifyKindSet)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfter, beforeAndAfter)
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAndAfter)
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NONE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAndAfter)
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfter, beforeAndAfter)
+ }
+ } else {
+ Seq(UpdateKindTrait.NONE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NONE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNone(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ createNewNode(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require nothing about UpdateKind.
+ val children = visitChildren(rel, UpdateKindTrait.NONE)
+ createNewNode(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank, rank.partitionKey, rank.orderKey)
+ visitRankStrategies(rankStrategies, requiredTrait, rankStrategy => rank.copy(rankStrategy))
+
+ case sortLimit: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ sortLimit, ImmutableBitSet.of(), sortLimit.getCollation)
+ visitRankStrategies(
+ rankStrategies,
+ requiredTrait,
+ rankStrategy => sortLimit.copy(rankStrategy))
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNone(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ createNewNode(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = join.getInputs.zipWithIndex.map {
+ case (child, childOrdinal) =>
+ val physicalChild = child.asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(physicalChild)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNone(inputModifyKindSet)
+ } else {
+ onlyAfterOrNone(inputModifyKindSet)
+ }
+ this.visit(physicalChild, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ createNewNode(join, Some(children.flatten.toList), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ // so it requires nothing about UpdateKind
+ val newRightOption = this.visit(right, UpdateKindTrait.NONE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ createNewNode(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
Review comment:
Sorry, I forgot to implement this conclusion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407153250
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, ImmutableBitSet.of(), rank.getCollation)
+ // creates new SortLimit nodes for every applied RankStrategy
+ val newSortLimitNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newSortLimitNodes, requiredTrait)
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ replaceChildrenAndTrait(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = List(0, 1).map { childOrdinal =>
+ val child = join.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(child)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNoUpdate(inputModifyKindSet)
+ } else {
+ onlyAfterOrNoUpdate(inputModifyKindSet)
+ }
+ this.visit(child, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ replaceChildrenAndTrait(join, Some(children.flatten), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ val newRightOption = this.visit(right, UpdateKindTrait.NO_UPDATE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ visitChildren(rel, requiredTrait) match {
+ case None => None
+ case Some(children) =>
+ val childTrait = children.head.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(rel, Some(children), childTrait)
+ }
+
+ case union: StreamExecUnion =>
+ val children = union.getInputs.map { case child: StreamPhysicalRel =>
+ val childModifyKindSet = getModifyKindSet(child)
+ val requiredChildTrait = if (childModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ requiredTrait
+ }
+ this.visit(child, requiredChildTrait)
+ }.toList
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ val updateKinds = children.flatten
+ .map(_.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE))
+ // union can just forward changes, can't actively satisfy to another changelog mode
+ val providedTrait = if (updateKinds.forall(k => UpdateKindTrait.NO_UPDATE == k)) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ updateKinds.head
+ }
+ replaceChildrenAndTrait(union, Some(children.flatten), providedTrait)
+
+ }
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ replaceChildrenAndTrait(rel, Some(List()), UpdateKindTrait.NO_UPDATE)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = if (scan.intermediateTable.isUpdateBeforeRequired) {
+ // we can't drop UPDATE_BEFORE if it is required by other parent blocks
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else {
+ requiredTrait
+ }
+ if (!providedTrait.satisfies(requiredTrait)) {
+ // require ONLY_AFTER but can only provide BEFORE_AND_AFTER
+ None
+ } else {
+ replaceChildrenAndTrait(rel, Some(List()), providedTrait)
+ }
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: UpdateKindTrait): Option[List[StreamPhysicalRel]] = {
+ val newChildren = for (child <- parent.getInputs) yield {
+ this.visit(child.asInstanceOf[StreamPhysicalRel], requiredChildrenTrait) match {
+ case None =>
+ // return None if one of the children can't satisfy
+ return None
+ case Some(newChild) =>
+ val providedTrait = newChild.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ val childDescription = newChild.getRelDetailedDescription
+ if (!providedTrait.satisfies(requiredChildrenTrait)) {
+ throw new TableException(s"Provided trait $providedTrait can't satisfy " +
+ s"required trait $requiredChildrenTrait. " +
+ s"This is a bug in planner, please file an issue. \n" +
+ s"Current node is $childDescription")
+ }
+ newChild
+ }
+ }
+ Some(newChildren.toList)
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ childrenOption: Option[List[StreamPhysicalRel]],
+ providedTrait: UpdateKindTrait): Option[StreamPhysicalRel] = childrenOption match {
+ case None =>
+ None
+ case Some(children) =>
+ val modifyKindSetTrait = node.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ val nodeDescription = node.getRelDetailedDescription
+ if (!modifyKindSetTrait.satisfies(providedTrait)) {
+ throw new TableException(s"ModifyKindSetTrait $modifyKindSetTrait can't satisfy " +
+ s"UpdateKindTrait $providedTrait. " +
+ s"This is a bug in planner, please file an issue. \n" +
+ s"Current node is $nodeDescription.")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ Some(node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel])
+ }
+
+ private def visitRankStrategies(
+ rankStrategies: Seq[RankProcessStrategy],
+ rankNodes: Seq[StreamPhysicalRel],
+ requiredUpdateKindTrait: UpdateKindTrait): Option[StreamPhysicalRel] = {
+ // go pass every RankProcessStrategy and Rank node, return the first satisfied converted node
+ rankStrategies.zip(rankNodes).flatMap {
Review comment:
current implementation needs to visit all RankStrategies, actually we only need the first satisfied one. so we can use `for` to iterator all RankStrategies and return when meet first satisfied one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407154080
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, ImmutableBitSet.of(), rank.getCollation)
+ // creates new SortLimit nodes for every applied RankStrategy
+ val newSortLimitNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newSortLimitNodes, requiredTrait)
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ replaceChildrenAndTrait(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = List(0, 1).map { childOrdinal =>
+ val child = join.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(child)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNoUpdate(inputModifyKindSet)
+ } else {
+ onlyAfterOrNoUpdate(inputModifyKindSet)
+ }
+ this.visit(child, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ replaceChildrenAndTrait(join, Some(children.flatten), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ val newRightOption = this.visit(right, UpdateKindTrait.NO_UPDATE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ visitChildren(rel, requiredTrait) match {
+ case None => None
+ case Some(children) =>
+ val childTrait = children.head.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(rel, Some(children), childTrait)
+ }
+
+ case union: StreamExecUnion =>
+ val children = union.getInputs.map { case child: StreamPhysicalRel =>
Review comment:
nit: single line for `case child: StreamPhysicalRel =>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* b8ea858f10d83717561e894c71e5b0b49fbd85f4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160141637) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417)
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160237416) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6813ce29fd9023e8cba34f9539ed09b080401ba1 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159519229) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248)
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407202237
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.xml
##########
@@ -313,17 +313,17 @@ LogicalProject(a1=[$1], b1=[$3])
</Resource>
<Resource name="planAfter">
<![CDATA[
-Join(joinType=[InnerJoin], where=[=(a1, b1)], select=[a1, b1], leftInputSpec=[JoinKeyContainsUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey], updateAsRetraction=[false], accMode=[Acc])
-:- Exchange(distribution=[hash[a1]], updateAsRetraction=[false], accMode=[Acc])
-: +- GroupAggregate(groupBy=[a1], select=[a1], updateAsRetraction=[false], accMode=[Acc])
-: +- Exchange(distribution=[hash[a1]], updateAsRetraction=[true], accMode=[Acc])
-: +- Calc(select=[a1, a2], updateAsRetraction=[true], accMode=[Acc])
-: +- TableSourceScan(table=[[default_catalog, default_database, A, source: [TestTableSource(a1, a2, a3)]]], fields=[a1, a2, a3], updateAsRetraction=[true], accMode=[Acc])
-+- Exchange(distribution=[hash[b1]], updateAsRetraction=[false], accMode=[Acc])
- +- GroupAggregate(groupBy=[b1], select=[b1], updateAsRetraction=[false], accMode=[Acc])
- +- Exchange(distribution=[hash[b1]], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[b1, b2], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, B, source: [TestTableSource(b1, b2, b3)]]], fields=[b1, b2, b3], updateAsRetraction=[true], accMode=[Acc])
+Join(joinType=[InnerJoin], where=[=(a1, b1)], select=[a1, b1], leftInputSpec=[JoinKeyContainsUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey], changelogMode=[I,UA,D])
Review comment:
`DELETE` is unnecessary ? `a1` and `b1` are `pk`s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160237416) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473)
* ce56fcd7c4168f70b239f3d62962e5b0146ad5d0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407201062
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/RetractionRulesWithTwoStageAggTest.xml
##########
@@ -39,19 +39,19 @@ LogicalAggregate(group=[{0}], frequency=[COUNT($0)])
</Resource>
<Resource name="planAfter">
<![CDATA[
-GlobalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(count$0) AS frequency], updateAsRetraction=[false], accMode=[Acc])
-+- Exchange(distribution=[hash[cnt]], updateAsRetraction=[true], accMode=[Acc])
- +- LocalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS count$0, COUNT_RETRACT(*) AS count1$1], updateAsRetraction=[true], accMode=[Acc])
- +- Union(all=[true], union=[cnt], updateAsRetraction=[true], accMode=[AccRetract])
- :- Calc(select=[CAST(cnt) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- GlobalGroupAggregate(groupBy=[word], select=[word, COUNT(count$0) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- Exchange(distribution=[hash[word]], updateAsRetraction=[true], accMode=[Acc])
- : +- LocalGroupAggregate(groupBy=[word], select=[word, COUNT(number) AS count$0], updateAsRetraction=[true], accMode=[Acc])
- : +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(word, number)]]], fields=[word, number], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[cnt], updateAsRetraction=[true], accMode=[Acc])
- +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, MyTable2, source: [TestTableSource(word, cnt)]]], fields=[word, cnt], updateAsRetraction=[true], accMode=[Acc])
+GlobalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(count$0) AS frequency], changelogMode=[I,UA,D])
++- Exchange(distribution=[hash[cnt]], changelogMode=[I])
+ +- LocalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS count$0, COUNT_RETRACT(*) AS count1$1], changelogMode=[I])
Review comment:
the changelog mode's value is incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407303722
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/UpdateKindTrait.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+/**
+ * UpdateKindTrait is used to describe the kind of update operation.
+ */
+class UpdateKindTrait(val updateKind: UpdateKind) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: UpdateKindTrait =>
+ // should totally match
+ other.updateKind == this.updateKind
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = UpdateKindTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = updateKind.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: UpdateKindTrait => this.updateKind.equals(t.updateKind)
+ case _ => false
+ }
+
+ override def toString: String = s"[${updateKind.toString}]"
+}
+
+object UpdateKindTrait {
+ val NO_UPDATE = new UpdateKindTrait(UpdateKind.NO_UPDATE)
Review comment:
`NO_UPDATE` is not just a placeholder here. For `AppendTableSink` and window aggregation, it requires children should produce `NO_UPDATE` trait. It doesnt make sense to require a `ONLY_UPDATE_AFTER` or `BEFORE_AND_AFTER` in this case.
Merging `ONLY_UPDATE_AFTER` and `BEFORE_AND_AFTER` into `ModifyKind.UPDATE` maybe confusing. Becasue a default `BEFORE_AND_AFTER` maybe wrong in some cases (e.g. upsert source produce `ONLY_AFTER`). Separating into 2 traits in on purpose, and I think it makes things cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159412901) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236)
* 6813ce29fd9023e8cba34f9539ed09b080401ba1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407963289
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,626 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNone, onlyAfterOrNone}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical node.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NONE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node's behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ createNewNode(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ createNewNode(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ createNewNode(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ createNewNode(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ createNewNode(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank and SortLimit supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ createNewNode(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ createNewNode(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ createNewNode(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ createNewNode(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (innerOrSemi) {
+ // forward left and right modify operations
+ new ModifyKindSetTrait(leftKindSet.union(rightKindSet))
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ createNewNode(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ createNewNode(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ createNewNode(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ createNewNode(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ createNewNode(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ createNewNode(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def createNewNode(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.minus(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfies required traits by input nodes of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val onlyAfter = onlyAfterOrNone(childModifyKindSet)
+ val beforeAndAfter = beforeAfterOrNone(childModifyKindSet)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfter, beforeAndAfter)
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAndAfter)
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NONE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAndAfter)
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfter, beforeAndAfter)
+ }
+ } else {
+ Seq(UpdateKindTrait.NONE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NONE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNone(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ createNewNode(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require nothing about UpdateKind.
+ val children = visitChildren(rel, UpdateKindTrait.NONE)
+ createNewNode(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank, rank.partitionKey, rank.orderKey)
+ visitRankStrategies(rankStrategies, requiredTrait, rankStrategy => rank.copy(rankStrategy))
+
+ case sortLimit: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ sortLimit, ImmutableBitSet.of(), sortLimit.getCollation)
+ visitRankStrategies(
+ rankStrategies,
+ requiredTrait,
+ rankStrategy => sortLimit.copy(rankStrategy))
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNone(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ createNewNode(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = join.getInputs.zipWithIndex.map {
+ case (child, childOrdinal) =>
+ val physicalChild = child.asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(physicalChild)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNone(inputModifyKindSet)
+ } else {
+ onlyAfterOrNone(inputModifyKindSet)
+ }
+ this.visit(physicalChild, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ createNewNode(join, Some(children.flatten.toList), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ // so it requires nothing about UpdateKind
+ val newRightOption = this.visit(right, UpdateKindTrait.NONE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ createNewNode(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
Review comment:
IIRC we've decided to not let `filter` passing the update kind. To solve the bad case like a single 'x < 10' condition which won't satisfy the `only_update_before` case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407134044
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ChangelogPlanUtils.scala
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils
+
+import org.apache.flink.table.connector.ChangelogMode
+import org.apache.flink.table.planner.plan.`trait`.{ModifyKind, ModifyKindSetTraitDef, UpdateKind, UpdateKindTraitDef}
+import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalRel
+import org.apache.flink.table.planner.plan.optimize.program.FlinkChangelogModeInferenceProgram
+import org.apache.flink.types.RowKind
+import org.apache.calcite.rel.RelNode
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Utilities for changelog plan.
+ */
+object ChangelogPlanUtils {
+
+ /**
+ * Returns true if the plan produces insert-only changes.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def isInsertOnly(plan: RelNode): Boolean = {
+ assert(plan.isInstanceOf[StreamPhysicalRel])
Review comment:
remove `assert`, the parameter should be `StreamPhysicalRel` directly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407351865
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTrait.scala
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+import scala.collection.JavaConversions._
+
+/**
+ * ModifyKindSetTrait is used to describe what modify operation will be produced by this node.
+ */
+class ModifyKindSetTrait(val modifyKindSet: ModifyKindSet) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: ModifyKindSetTrait =>
+ // it’s satisfied when modify kinds are included in the required set,
+ // e.g. [I,U] satisfy [I,U,D]
+ // [I,U,D] not satisfy [I,D]
+ this.modifyKindSet.getContainedKinds.forall(other.modifyKindSet.contains)
+ case other: UpdateKindTrait =>
Review comment:
why `ModifyKindSetTrait` will be compared with `UpdateKindTrait`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407152742
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
Review comment:
move `val newRankNodes = rankStrategies.map(rank.copy)` into `visitRankStrategies ` method. which means we apply all RankStrategys to current node
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 6d34187c1dde98f1fc8573063e298bd481224688 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160017412) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384)
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160352250",
"triggerID" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"triggerType" : "PUSH"
}, {
"hash" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7503",
"triggerID" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160237416) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473)
* ce56fcd7c4168f70b239f3d62962e5b0146ad5d0 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160352250) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7503)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159412901) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407133388
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/DagOptimizationTest.xml
##########
@@ -959,15 +951,15 @@ LogicalSink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b,
</Resource>
<Resource name="planAfter">
<![CDATA[
-Sink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
-+- Union(all=[true], union=[b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
- :- GroupAggregate(groupBy=[b, c], select=[b, c, SUM(a) AS a_sum], updateAsRetraction=[false], accMode=[Acc])
- : +- Exchange(distribution=[hash[b, c]], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]], fields=[a, b, c], updateAsRetraction=[true], accMode=[Acc], reuse_id=[1])
- +- Calc(select=[CAST(1:BIGINT) AS b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
- +- GroupAggregate(groupBy=[c], select=[c, SUM(a) AS a_sum], updateAsRetraction=[false], accMode=[Acc])
- +- Exchange(distribution=[hash[c]], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[c, a], updateAsRetraction=[true], accMode=[Acc])
+Sink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b, c, a_sum], changelogMode=[NONE])
Review comment:
It's better the pk info can also be printed for upsert sink.
just for this case, the pk of upsert sink is empty(is illegal?), `UA` is needed ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407146140
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
Review comment:
nit: "Rank" -> "Rank and SortLimit"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407080113
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTrait.scala
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+import scala.collection.JavaConversions._
+
+/**
+ * ModifyKindSetTrait is used to describe what modify operation will be produced by this node.
+ */
+class ModifyKindSetTrait(val modifyKindSet: ModifyKindSet) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: ModifyKindSetTrait =>
+ // it’s satisfied when modify kinds are included in the required set,
+ // e.g. [I,U] satisfy [I,U,D]
+ // [I,U,D] not satisfy [I,D]
+ this.modifyKindSet.getContainedKinds.forall(other.modifyKindSet.contains)
+ case other: UpdateKindTrait =>
+ // it's satisfied when UpdateKind matches ModifyKindSet
+ other.updateKind match {
+ case UpdateKind.NO_UPDATE =>
+ modifyKindSet.containsOnly(ModifyKind.INSERT)
+ case UpdateKind.BEFORE_AND_AFTER | UpdateKind.ONLY_UPDATE_AFTER =>
+ modifyKindSet.contains(ModifyKind.UPDATE)
+ }
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = ModifyKindSetTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = modifyKindSet.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: ModifyKindSetTrait => this.modifyKindSet.equals(t.modifyKindSet)
+ case _ => false
+ }
+
+ override def toString: String = s"[${modifyKindSet.toString}]"
+}
+
+object ModifyKindSetTrait {
+ /**
+ * An empty [[ModifyKindSetTrait]] which doesn't contain any [[ModifyKind]].
+ */
+ val EMPTY = new ModifyKindSetTrait(ModifyKindSet.newBuilder().build())
+
+ /**
+ * Insert-only [[ModifyKindSetTrait]].
+ */
+ val INSERT_ONLY = new ModifyKindSetTrait(ModifyKindSet.newBuilder
+ .addContainedKind(ModifyKind.INSERT)
+ .build)
+
+ /**
+ * A modify [[ModifyKindSetTrait]] contains all change operations.
+ */
+ val ALL_CHANGES = new ModifyKindSetTrait(ModifyKindSet.newBuilder
+ .addContainedKind(ModifyKind.INSERT)
Review comment:
we can use `ModifyKindSet.ALL_CHANGES` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 6d34187c1dde98f1fc8573063e298bd481224688 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160017412) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384)
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6813ce29fd9023e8cba34f9539ed09b080401ba1 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159519229) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248)
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160066338) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409)
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* b8ea858f10d83717561e894c71e5b0b49fbd85f4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-613507721
Thanks for the reviewing @KurtYoung . I have updated the branch to not satisfy ONLY_UPDATE_BEFORE for filters.
And added tests in `SinkTest` and `TableSinkITCase`. Which should already fix FLINK-9528.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407079519
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/ModifyKindSet.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * The set of modify operations contained in a changelog.
+ *
+ * @see ModifyKind
+ */
+public class ModifyKindSet {
+
+ /**
+ * Insert-only modify kind set.
+ */
+ public static final ModifyKindSet INSERT_ONLY = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .build();
+
+ /**
+ * A modify kind set contains all change operations.
+ */
+ public static final ModifyKindSet ALL_CHANGES = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ .addContainedKind(ModifyKind.DELETE)
+ .build();
+
+ private final Set<ModifyKind> kinds;
+
+ private ModifyKindSet(Set<ModifyKind> kinds) {
+ this.kinds = Collections.unmodifiableSet(kinds);
+ }
+
+ public Set<ModifyKind> getContainedKinds() {
+ return kinds;
+ }
+
+ public boolean contains(ModifyKind kind) {
+ return kinds.contains(kind);
+ }
+
+ public boolean containsOnly(ModifyKind kind) {
+ return kinds.size() == 1 && kinds.contains(kind);
+ }
+
+ public boolean isInsertOnly() {
+ return containsOnly(ModifyKind.INSERT);
+ }
+
+ public int size() {
+ return kinds.size();
+ }
+
+ public boolean isEmpty() {
+ return kinds.isEmpty();
+ }
+
+ /**
+ * Returns a new set of ModifyKind which is the difference between two sets.
+ * It is also equal to {@code this.kinds - that.kinds}. For example:
+ * [I,U,D] diff [I] = [U,D]
+ * [I,U] diff [U,D] = [I]
+ * [I,U,D] diff [I,U,D] = []
+ */
+ public ModifyKindSet diff(ModifyKindSet other) {
+ Set<ModifyKind> result = EnumSet.noneOf(ModifyKind.class);
+ result.addAll(this.kinds);
+ result.removeAll(other.kinds);
+ return new ModifyKindSet(result);
+ }
+
+ /**
+ * Returns a new ModifyKindSet with all kinds set in both this set and in another set.
+ */
+ public ModifyKindSet intersect(ModifyKindSet other) {
+ Builder builder = new Builder();
+ for (ModifyKind kind : other.getContainedKinds()) {
+ if (this.contains(kind)) {
+ builder.addContainedKind(kind);
+ }
+ }
+ return builder.build();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ModifyKindSet that = (ModifyKindSet) o;
+ return Objects.equals(kinds, that.kinds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(kinds);
+ }
+
+ @Override
+ public String toString() {
+ if (kinds.isEmpty()) {
+ return "NONE";
+ }
+ List<String> modifyKinds = new ArrayList<>();
+ if (contains(ModifyKind.INSERT)) {
+ modifyKinds.add("I");
+ }
+ if (contains(ModifyKind.UPDATE)) {
+ modifyKinds.add("U");
+ }
+ if (contains(ModifyKind.DELETE)) {
+ modifyKinds.add("D");
+ }
+ return String.join(",", modifyKinds);
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Returns the union of a number of ModifyKindSets.
+ */
+ public static ModifyKindSet union(ModifyKindSet... modifyKindSets) {
+ Builder builder = newBuilder();
+ for (ModifyKindSet set : modifyKindSets) {
+ for (ModifyKind kind : set.getContainedKinds()) {
+ builder.addContainedKind(kind);
+ }
+ }
+ return builder.build();
+ }
+
+ /**
+ * Builder for configuring and creating instances of {@link ModifyKindSet}.
+ */
+ public static Builder newBuilder() {
+ return new Builder();
+ }
+
+ /**
+ * Builder for configuring and creating instances of {@link ModifyKindSet}.
+ */
+ public static class Builder {
+
+ private final Set<ModifyKind> kinds = EnumSet.noneOf(ModifyKind.class);
+
+ public Builder() {
Review comment:
we can make this constructor private, since we already have a public `newBuilder` method for creating a `Builder`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407153603
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, ImmutableBitSet.of(), rank.getCollation)
+ // creates new SortLimit nodes for every applied RankStrategy
+ val newSortLimitNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newSortLimitNodes, requiredTrait)
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ replaceChildrenAndTrait(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = List(0, 1).map { childOrdinal =>
Review comment:
`join.getInputs.zipWithIndex`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159412901) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407154264
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, ImmutableBitSet.of(), rank.getCollation)
+ // creates new SortLimit nodes for every applied RankStrategy
+ val newSortLimitNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newSortLimitNodes, requiredTrait)
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ replaceChildrenAndTrait(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = List(0, 1).map { childOrdinal =>
+ val child = join.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(child)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNoUpdate(inputModifyKindSet)
+ } else {
+ onlyAfterOrNoUpdate(inputModifyKindSet)
+ }
+ this.visit(child, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ replaceChildrenAndTrait(join, Some(children.flatten), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ val newRightOption = this.visit(right, UpdateKindTrait.NO_UPDATE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ visitChildren(rel, requiredTrait) match {
+ case None => None
+ case Some(children) =>
+ val childTrait = children.head.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(rel, Some(children), childTrait)
+ }
+
+ case union: StreamExecUnion =>
+ val children = union.getInputs.map { case child: StreamPhysicalRel =>
+ val childModifyKindSet = getModifyKindSet(child)
+ val requiredChildTrait = if (childModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ requiredTrait
+ }
+ this.visit(child, requiredChildTrait)
+ }.toList
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ val updateKinds = children.flatten
+ .map(_.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE))
+ // union can just forward changes, can't actively satisfy to another changelog mode
+ val providedTrait = if (updateKinds.forall(k => UpdateKindTrait.NO_UPDATE == k)) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ updateKinds.head
Review comment:
union all `updateKinds ` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* b8ea858f10d83717561e894c71e5b0b49fbd85f4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160141637) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417)
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407300327
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/DagOptimizationTest.xml
##########
@@ -959,15 +951,15 @@ LogicalSink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b,
</Resource>
<Resource name="planAfter">
<![CDATA[
-Sink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
-+- Union(all=[true], union=[b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
- :- GroupAggregate(groupBy=[b, c], select=[b, c, SUM(a) AS a_sum], updateAsRetraction=[false], accMode=[Acc])
- : +- Exchange(distribution=[hash[b, c]], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]], fields=[a, b, c], updateAsRetraction=[true], accMode=[Acc], reuse_id=[1])
- +- Calc(select=[CAST(1:BIGINT) AS b, c, a_sum], updateAsRetraction=[false], accMode=[Acc])
- +- GroupAggregate(groupBy=[c], select=[c, SUM(a) AS a_sum], updateAsRetraction=[false], accMode=[Acc])
- +- Exchange(distribution=[hash[c]], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[c, a], updateAsRetraction=[true], accMode=[Acc])
+Sink(name=[`default_catalog`.`default_database`.`upsertSink`], fields=[b, c, a_sum], changelogMode=[NONE])
Review comment:
Since FLIP-95 new sink interfaces, PK of sink is defined in DDL, not derived from query anymore. Deriving primary key from query confused many users and make Flink SQL not flexible. So I will add the PK information once we integrate with new sink interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407147199
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
Review comment:
extract `onlyAfterOrNoUpdate(childModifyKindSet)` as a variable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407331685
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/RetractionRulesTest.xml
##########
@@ -39,15 +39,15 @@ LogicalAggregate(group=[{0}], frequency=[COUNT($0)])
</Resource>
<Resource name="planAfter">
<![CDATA[
-GroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS frequency], updateAsRetraction=[false], accMode=[Acc])
-+- Exchange(distribution=[hash[cnt]], updateAsRetraction=[true], accMode=[AccRetract])
- +- Union(all=[true], union=[cnt], updateAsRetraction=[true], accMode=[AccRetract])
- :- Calc(select=[CAST(cnt) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- GroupAggregate(groupBy=[word], select=[word, COUNT(number) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- Exchange(distribution=[hash[word]], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(word, number)]]], fields=[word, number], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[cnt], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, MyTable2, source: [TestTableSource(word, cnt)]]], fields=[word, cnt], updateAsRetraction=[true], accMode=[Acc])
+GroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS frequency], changelogMode=[I,UA,D])
Review comment:
I merged `RetractionRulesTest` and `RetractionRulesWithTwoStageAggTest` into `ChangelogModeInferenceTest` using `Parameterized`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407348957
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/ModifyKindSet.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * The set of modify operations contained in a changelog.
+ *
+ * @see ModifyKind
+ */
+public class ModifyKindSet {
+
+ /**
+ * Insert-only modify kind set.
+ */
+ public static final ModifyKindSet INSERT_ONLY = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .build();
+
+ /**
+ * A modify kind set contains all change operations.
+ */
+ public static final ModifyKindSet ALL_CHANGES = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ .addContainedKind(ModifyKind.DELETE)
+ .build();
+
+ private final Set<ModifyKind> kinds;
+
+ private ModifyKindSet(Set<ModifyKind> kinds) {
+ this.kinds = Collections.unmodifiableSet(kinds);
+ }
+
+ public Set<ModifyKind> getContainedKinds() {
+ return kinds;
+ }
+
+ public boolean contains(ModifyKind kind) {
+ return kinds.contains(kind);
+ }
+
+ public boolean containsOnly(ModifyKind kind) {
+ return kinds.size() == 1 && kinds.contains(kind);
+ }
+
+ public boolean isInsertOnly() {
+ return containsOnly(ModifyKind.INSERT);
+ }
+
+ public int size() {
+ return kinds.size();
+ }
+
+ public boolean isEmpty() {
+ return kinds.isEmpty();
+ }
+
+ /**
+ * Returns a new set of ModifyKind which is the difference between two sets.
+ * It is also equal to {@code this.kinds - that.kinds}. For example:
+ * [I,U,D] diff [I] = [U,D]
+ * [I,U] diff [U,D] = [I]
+ * [I,U,D] diff [I,U,D] = []
+ */
+ public ModifyKindSet diff(ModifyKindSet other) {
+ Set<ModifyKind> result = EnumSet.noneOf(ModifyKind.class);
+ result.addAll(this.kinds);
+ result.removeAll(other.kinds);
+ return new ModifyKindSet(result);
+ }
+
+ /**
+ * Returns a new ModifyKindSet with all kinds set in both this set and in another set.
+ */
+ public ModifyKindSet intersect(ModifyKindSet other) {
+ Builder builder = new Builder();
+ for (ModifyKind kind : other.getContainedKinds()) {
+ if (this.contains(kind)) {
+ builder.addContainedKind(kind);
+ }
+ }
+ return builder.build();
+ }
+
+ /**
+ * Returns a new ModifyKindSet with the union of the other ModifyKindSet.
+ */
+ public ModifyKindSet union(ModifyKindSet other) {
+ return union(this, other);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ModifyKindSet that = (ModifyKindSet) o;
+ return Objects.equals(kinds, that.kinds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(kinds);
+ }
+
+ @Override
+ public String toString() {
+ if (kinds.isEmpty()) {
Review comment:
I think we should protect this class to let it be always non-empty.
Does an empty `ModifyKindSet` make any sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407497035
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
Review comment:
nit: current node's behavior
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407348519
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/ModifyKindSet.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * The set of modify operations contained in a changelog.
+ *
+ * @see ModifyKind
+ */
+public class ModifyKindSet {
+
+ /**
+ * Insert-only modify kind set.
+ */
+ public static final ModifyKindSet INSERT_ONLY = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .build();
+
+ /**
+ * A modify kind set contains all change operations.
+ */
+ public static final ModifyKindSet ALL_CHANGES = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ .addContainedKind(ModifyKind.DELETE)
+ .build();
+
+ private final Set<ModifyKind> kinds;
+
+ private ModifyKindSet(Set<ModifyKind> kinds) {
+ this.kinds = Collections.unmodifiableSet(kinds);
+ }
+
+ public Set<ModifyKind> getContainedKinds() {
+ return kinds;
+ }
+
+ public boolean contains(ModifyKind kind) {
+ return kinds.contains(kind);
+ }
+
+ public boolean containsOnly(ModifyKind kind) {
+ return kinds.size() == 1 && kinds.contains(kind);
+ }
+
+ public boolean isInsertOnly() {
+ return containsOnly(ModifyKind.INSERT);
+ }
+
+ public int size() {
+ return kinds.size();
+ }
+
+ public boolean isEmpty() {
+ return kinds.isEmpty();
+ }
+
+ /**
+ * Returns a new set of ModifyKind which is the difference between two sets.
+ * It is also equal to {@code this.kinds - that.kinds}. For example:
+ * [I,U,D] diff [I] = [U,D]
+ * [I,U] diff [U,D] = [I]
+ * [I,U,D] diff [I,U,D] = []
+ */
+ public ModifyKindSet diff(ModifyKindSet other) {
Review comment:
use `minus`? `diff` sounds like another meaning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160352250",
"triggerID" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"triggerType" : "PUSH"
}, {
"hash" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7503",
"triggerID" : "ce56fcd7c4168f70b239f3d62962e5b0146ad5d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* ce56fcd7c4168f70b239f3d62962e5b0146ad5d0 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160352250) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7503)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59ab8cedf26f3b55602b448951fa18e8e7997bb5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159526393) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253)
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 6d34187c1dde98f1fc8573063e298bd481224688 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159412901) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407199772
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/SinkTest.xml
##########
@@ -16,27 +16,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
<Root>
- <TestCase name="testAppendSink">
Review comment:
add a case to test append sink ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160066338) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409)
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407349672
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/UpdateKind.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+/**
+ * Lists all kinds of {@link ModifyKind#UPDATE} operation.
+ */
+public enum UpdateKind {
+
+ /**
+ * No update operation.
+ */
+ NO_UPDATE,
+
+ /**
+ * This kind indicates that operators should emit update changes just as a row of
+ * {@code RowKind#UPDATE_AFTER}.
+ */
+ ONLY_UPDATE_AFTER,
+
+ /**
+ * This kind indicates that operators should emit update changes in the way that
+ * a row of {@code RowKind#UPDATE_BEFORE} and a row of {@code RowKind#UPDATE_AFTER} together.
+ */
+ BEFORE_AND_AFTER;
+
+ /**
+ * Return a new UpdateKind which merges this UpdateKind and the other UpdateKind.
+ */
+ public UpdateKind merge(UpdateKind that) {
Review comment:
It's hard to imagine & understand why this class support `merge`, what should be the expected behavior?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407137103
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/UpdateKindTrait.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+/**
+ * UpdateKindTrait is used to describe the kind of update operation.
+ */
+class UpdateKindTrait(val updateKind: UpdateKind) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: UpdateKindTrait =>
+ // should totally match
+ other.updateKind == this.updateKind
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = UpdateKindTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = updateKind.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: UpdateKindTrait => this.updateKind.equals(t.updateKind)
+ case _ => false
+ }
+
+ override def toString: String = s"[${updateKind.toString}]"
+}
+
+object UpdateKindTrait {
+ val NO_UPDATE = new UpdateKindTrait(UpdateKind.NO_UPDATE)
Review comment:
Now a complete change mode is represented by two `Trait`s, `NO_UPDATE` is just a placeholder which also means `ModifyKindSetTrait` does not contain `UPDATE`.
Another idea is that we can treat `ONLY_UPDATE_AFTER` and `BEFORE_AND_AFTER` as a sub-attribute of `ModifyKind.UPDATE` (`BEFORE_AND_AFTER` is default), then we can use only one trait to represent changelog mode. The optimization we want to apply is whether `BEFORE_AND_AFTER` can be changed to `ONLY_UPDATE_AFTER`.
It's better we can also give some comments here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160237416) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r408223133
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/api/stream/ExplainTest.xml
##########
@@ -45,11 +45,11 @@ LogicalProject(EXPR$0=[$1])
+- LogicalTableScan(table=[[default_catalog, default_database, MyTable1]])
== Optimized Logical Plan ==
-Calc(select=[EXPR$0], updateAsRetraction=[false], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
-+- GroupAggregate(groupBy=[a], select=[a, COUNT(*) AS EXPR$0], updateAsRetraction=[false], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- Exchange(distribution=[hash[a]], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- Calc(select=[a], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
- +- DataStreamScan(table=[[default_catalog, default_database, MyTable1]], fields=[a, b, c], updateAsRetraction=[true], accMode=[Acc]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
+Calc(select=[EXPR$0], changelogMode=[I,UA]): rowcount = , cumulative cost = {rows, cpu, io, network, memory}
Review comment:
Because we are using `ONLY_UPDATE_AFTER` as the default required trait for queries without sink. This is set in [StreamCommonSubGraphBasedOptimizer.scala#L60](https://github.com/wuchong/flink/blob/9dcfd29d468b25f245e7b178ea4ae1a331ea6b39/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/StreamCommonSubGraphBasedOptimizer.scala#L60) and passed by `StreamOptimizeContext#isUpdateBeforeRequired`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407138288
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.scala
##########
@@ -75,7 +74,7 @@ object ExecNodePlanDumper {
* @param nodes the ExecNodes to convert
* @param detailLevel detailLevel defines detail levels for EXPLAIN PLAN.
* @param withExecNodeId whether including ID of ExecNode
- * @param withRetractTraits whether including Retraction Traits of RelNode corresponding to
+ * @param withChangelogTraits whether including Retraction Traits of RelNode corresponding to
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407138222
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ExecNodePlanDumper.scala
##########
@@ -48,7 +47,7 @@ object ExecNodePlanDumper {
* @param node the ExecNode to convert
* @param detailLevel detailLevel defines detail levels for EXPLAIN PLAN.
* @param withExecNodeId whether including ID of ExecNode
- * @param withRetractTraits whether including Retraction Traits of RelNode corresponding to
+ * @param withChangelogTraits whether including Retraction Traits of RelNode corresponding to
Review comment:
`Retraction Traits` -> `Changelog Traits`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407329570
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/RetractionRulesWithTwoStageAggTest.xml
##########
@@ -39,19 +39,19 @@ LogicalAggregate(group=[{0}], frequency=[COUNT($0)])
</Resource>
<Resource name="planAfter">
<![CDATA[
-GlobalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(count$0) AS frequency], updateAsRetraction=[false], accMode=[Acc])
-+- Exchange(distribution=[hash[cnt]], updateAsRetraction=[true], accMode=[Acc])
- +- LocalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS count$0, COUNT_RETRACT(*) AS count1$1], updateAsRetraction=[true], accMode=[Acc])
- +- Union(all=[true], union=[cnt], updateAsRetraction=[true], accMode=[AccRetract])
- :- Calc(select=[CAST(cnt) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- GlobalGroupAggregate(groupBy=[word], select=[word, COUNT(count$0) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- Exchange(distribution=[hash[word]], updateAsRetraction=[true], accMode=[Acc])
- : +- LocalGroupAggregate(groupBy=[word], select=[word, COUNT(number) AS count$0], updateAsRetraction=[true], accMode=[Acc])
- : +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(word, number)]]], fields=[word, number], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[cnt], updateAsRetraction=[true], accMode=[Acc])
- +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, MyTable2, source: [TestTableSource(word, cnt)]]], fields=[word, cnt], updateAsRetraction=[true], accMode=[Acc])
+GlobalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(count$0) AS frequency], changelogMode=[I,UA,D])
++- Exchange(distribution=[hash[cnt]], changelogMode=[I])
+ +- LocalGroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS count$0, COUNT_RETRACT(*) AS count1$1], changelogMode=[I])
Review comment:
It is as expected. LocalAggregate will emit accumulators to global aggregates. All the accumulators are encoded in insert messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160066338) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409)
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159412901) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236)
* 6813ce29fd9023e8cba34f9539ed09b080401ba1 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159519229) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-612773432
Thanks for the detailed review @godfreyhe and @libenchao . I have addressed all the comments.
Please have a look again when you are free.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407080060
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTrait.scala
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+import scala.collection.JavaConversions._
+
+/**
+ * ModifyKindSetTrait is used to describe what modify operation will be produced by this node.
+ */
+class ModifyKindSetTrait(val modifyKindSet: ModifyKindSet) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: ModifyKindSetTrait =>
+ // it’s satisfied when modify kinds are included in the required set,
+ // e.g. [I,U] satisfy [I,U,D]
+ // [I,U,D] not satisfy [I,D]
+ this.modifyKindSet.getContainedKinds.forall(other.modifyKindSet.contains)
+ case other: UpdateKindTrait =>
+ // it's satisfied when UpdateKind matches ModifyKindSet
+ other.updateKind match {
+ case UpdateKind.NO_UPDATE =>
+ modifyKindSet.containsOnly(ModifyKind.INSERT)
+ case UpdateKind.BEFORE_AND_AFTER | UpdateKind.ONLY_UPDATE_AFTER =>
+ modifyKindSet.contains(ModifyKind.UPDATE)
+ }
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = ModifyKindSetTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = modifyKindSet.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: ModifyKindSetTrait => this.modifyKindSet.equals(t.modifyKindSet)
+ case _ => false
+ }
+
+ override def toString: String = s"[${modifyKindSet.toString}]"
+}
+
+object ModifyKindSetTrait {
+ /**
+ * An empty [[ModifyKindSetTrait]] which doesn't contain any [[ModifyKind]].
+ */
+ val EMPTY = new ModifyKindSetTrait(ModifyKindSet.newBuilder().build())
+
+ /**
+ * Insert-only [[ModifyKindSetTrait]].
+ */
+ val INSERT_ONLY = new ModifyKindSetTrait(ModifyKindSet.newBuilder
+ .addContainedKind(ModifyKind.INSERT)
Review comment:
we can use `ModifyKindSet.INSERT_ONLY` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407201076
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/RetractionRulesWithTwoStageAggTest.xml
##########
@@ -68,12 +68,12 @@ LogicalProject(EXPR$0=[$1])
</Resource>
<Resource name="planAfter">
<![CDATA[
-Calc(select=[EXPR$0], updateAsRetraction=[false], accMode=[Acc])
-+- GlobalGroupAggregate(groupBy=[word], select=[word, COUNT(count$0) AS EXPR$0], updateAsRetraction=[false], accMode=[Acc])
- +- Exchange(distribution=[hash[word]], updateAsRetraction=[true], accMode=[Acc])
- +- LocalGroupAggregate(groupBy=[word], select=[word, COUNT(number) AS count$0], updateAsRetraction=[true], accMode=[Acc])
- +- MiniBatchAssigner(interval=[1000ms], mode=[ProcTime], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(word, number)]]], fields=[word, number], updateAsRetraction=[true], accMode=[Acc])
+Calc(select=[EXPR$0], changelogMode=[I,UA])
++- GlobalGroupAggregate(groupBy=[word], select=[word, COUNT(count$0) AS EXPR$0], changelogMode=[I,UA])
+ +- Exchange(distribution=[hash[word]], changelogMode=[I])
+ +- LocalGroupAggregate(groupBy=[word], select=[word, COUNT(number) AS count$0], changelogMode=[I])
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407201224
##########
File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/RetractionRulesTest.xml
##########
@@ -39,15 +39,15 @@ LogicalAggregate(group=[{0}], frequency=[COUNT($0)])
</Resource>
<Resource name="planAfter">
<![CDATA[
-GroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS frequency], updateAsRetraction=[false], accMode=[Acc])
-+- Exchange(distribution=[hash[cnt]], updateAsRetraction=[true], accMode=[AccRetract])
- +- Union(all=[true], union=[cnt], updateAsRetraction=[true], accMode=[AccRetract])
- :- Calc(select=[CAST(cnt) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- GroupAggregate(groupBy=[word], select=[word, COUNT(number) AS cnt], updateAsRetraction=[true], accMode=[AccRetract])
- : +- Exchange(distribution=[hash[word]], updateAsRetraction=[true], accMode=[Acc])
- : +- TableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(word, number)]]], fields=[word, number], updateAsRetraction=[true], accMode=[Acc])
- +- Calc(select=[cnt], updateAsRetraction=[true], accMode=[Acc])
- +- TableSourceScan(table=[[default_catalog, default_database, MyTable2, source: [TestTableSource(word, cnt)]]], fields=[word, cnt], updateAsRetraction=[true], accMode=[Acc])
+GroupAggregate(groupBy=[cnt], select=[cnt, COUNT_RETRACT(cnt) AS frequency], changelogMode=[I,UA,D])
Review comment:
the names of `RetractionRulesTest` and `RetractionRulesWithTwoStageAggTest` should also be updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong merged pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong merged pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407155058
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/RelNodeBlock.scala
##########
@@ -146,14 +153,30 @@ class RelNodeBlock(val outputNode: RelNode) {
def getOptimizedPlan: RelNode = optimizedPlan.orNull
- def setUpdateAsRetraction(updateAsRetract: Boolean): Unit = {
- // set child block updateAsRetract, a child may have multi father.
- if (updateAsRetract) {
- this.updateAsRetract = true
+ // TODO: not needed?
+ def setRequiredModifyKindSet(modifyKindSet: ModifyKindSet, requester: String): Unit = {
Review comment:
remove this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160066338) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409)
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407199265
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/TraitUtil.scala
##########
@@ -57,12 +57,4 @@ object TraitUtil {
}
}
- /**
- * Checks if a [[RelNode]] is in [[AccMode.AccRetract]] mode.
- */
- def isAccRetract(rel: RelNode): Boolean = {
Review comment:
remove useless import too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407196544
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamExecIntermediateTableScan.scala
##########
@@ -39,18 +39,6 @@ class StreamExecIntermediateTableScan(
extends CommonIntermediateTableScan(cluster, traitSet, table)
with StreamPhysicalRel {
- def isAccRetract: Boolean = intermediateTable.isAccRetract
-
- override def producesUpdates: Boolean = {
- !UpdatingPlanChecker.isAppendOnly(intermediateTable.relNode)
Review comment:
remove useless import too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407196427
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamExecGroupAggregate.scala
##########
@@ -64,7 +63,14 @@ class StreamExecGroupAggregate(
with StreamExecNode[BaseRow] {
val aggInfoList: AggregateInfoList = {
- val needRetraction = StreamExecRetractionRules.isAccRetract(getInput)
+ // need to call `retract()` if input contains update or delete
+ val modifyKindSetTrait = inputRel.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ val needRetraction = if (modifyKindSetTrait == null) {
+ // FlinkChangelogModeInferenceProgram is not applied yet, false as default
+ false
+ } else {
+ !modifyKindSetTrait.modifyKindSet.isInsertOnly
+ }
Review comment:
extract these code as a method, and put it in `AggregateUtil`, both `StreamExecGroupAggregate` and `StreamExecGroupTableAggregate` can use it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407080211
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTrait.scala
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+import scala.collection.JavaConversions._
+
+/**
+ * ModifyKindSetTrait is used to describe what modify operation will be produced by this node.
+ */
+class ModifyKindSetTrait(val modifyKindSet: ModifyKindSet) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: ModifyKindSetTrait =>
+ // it’s satisfied when modify kinds are included in the required set,
+ // e.g. [I,U] satisfy [I,U,D]
+ // [I,U,D] not satisfy [I,D]
+ this.modifyKindSet.getContainedKinds.forall(other.modifyKindSet.contains)
+ case other: UpdateKindTrait =>
+ // it's satisfied when UpdateKind matches ModifyKindSet
+ other.updateKind match {
+ case UpdateKind.NO_UPDATE =>
+ modifyKindSet.containsOnly(ModifyKind.INSERT)
+ case UpdateKind.BEFORE_AND_AFTER | UpdateKind.ONLY_UPDATE_AFTER =>
+ modifyKindSet.contains(ModifyKind.UPDATE)
+ }
+ case _ => false
+ }
+
+ override def getTraitDef: RelTraitDef[_ <: RelTrait] = ModifyKindSetTraitDef.INSTANCE
+
+ override def register(planner: RelOptPlanner): Unit = {}
+
+ override def hashCode(): Int = modifyKindSet.hashCode()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case t: ModifyKindSetTrait => this.modifyKindSet.equals(t.modifyKindSet)
+ case _ => false
+ }
+
+ override def toString: String = s"[${modifyKindSet.toString}]"
+}
+
+object ModifyKindSetTrait {
+ /**
+ * An empty [[ModifyKindSetTrait]] which doesn't contain any [[ModifyKind]].
+ */
+ val EMPTY = new ModifyKindSetTrait(ModifyKindSet.newBuilder().build())
+
+ /**
+ * Insert-only [[ModifyKindSetTrait]].
+ */
+ val INSERT_ONLY = new ModifyKindSetTrait(ModifyKindSet.newBuilder
+ .addContainedKind(ModifyKind.INSERT)
+ .build)
+
+ /**
+ * A modify [[ModifyKindSetTrait]] contains all change operations.
Review comment:
nit: A [[ModifyKindSetTrait]] which contains
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407340201
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ChangelogPlanUtils.scala
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils
+
+import org.apache.flink.table.connector.ChangelogMode
+import org.apache.flink.table.planner.plan.`trait`.{ModifyKind, ModifyKindSetTraitDef, UpdateKind, UpdateKindTraitDef}
+import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalRel
+import org.apache.flink.table.planner.plan.optimize.program.FlinkChangelogModeInferenceProgram
+import org.apache.flink.types.RowKind
+import org.apache.calcite.rel.RelNode
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Utilities for changelog plan.
+ */
+object ChangelogPlanUtils {
+
+ /**
+ * Returns true if the plan produces insert-only changes.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def isInsertOnly(plan: RelNode): Boolean = {
+ assert(plan.isInstanceOf[StreamPhysicalRel])
Review comment:
I still would like to keep this parameter as `RelNode`, otherwise, we will have a lot of duplicate codes, e.g. `getInput().asInstanceOf[StreamPhysicalNode]`. As this is an internal method, I think it's find to have an assert here to reduce the duplicate codes.
For other methods in this class, I changed to `StreamPhysicalRel` parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407152485
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
Review comment:
case sortLimit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6813ce29fd9023e8cba34f9539ed09b080401ba1 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159519229) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407212078
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
Review comment:
nit: "for every physical node." or "for all physical nodes."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407134101
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ChangelogPlanUtils.scala
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils
+
+import org.apache.flink.table.connector.ChangelogMode
+import org.apache.flink.table.planner.plan.`trait`.{ModifyKind, ModifyKindSetTraitDef, UpdateKind, UpdateKindTraitDef}
+import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalRel
+import org.apache.flink.table.planner.plan.optimize.program.FlinkChangelogModeInferenceProgram
+import org.apache.flink.types.RowKind
+import org.apache.calcite.rel.RelNode
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Utilities for changelog plan.
+ */
+object ChangelogPlanUtils {
+
+ /**
+ * Returns true if the plan produces insert-only changes.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def isInsertOnly(plan: RelNode): Boolean = {
+ assert(plan.isInstanceOf[StreamPhysicalRel])
+ val modifyKindSetTrait = plan.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ modifyKindSetTrait.modifyKindSet.isInsertOnly
+ }
+
+ /**
+ * Returns true if the [[RelNode]] will generate UPDATE_BEFORE messages.
+ * This method is used to determine whether the runtime operator should
+ * produce UPDATE_BEFORE messages with UPDATE_AFTER message together.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def generateUpdateBefore(node: RelNode): Boolean = {
+ assert(node.isInstanceOf[StreamPhysicalRel])
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407073916
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
Review comment:
nit: satisfies required traits by input nodes of current node.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407438440
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/ModifyKindSetTrait.scala
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTrait, RelTraitDef}
+
+import scala.collection.JavaConversions._
+
+/**
+ * ModifyKindSetTrait is used to describe what modify operation will be produced by this node.
+ */
+class ModifyKindSetTrait(val modifyKindSet: ModifyKindSet) extends RelTrait {
+
+ override def satisfies(relTrait: RelTrait): Boolean = relTrait match {
+ case other: ModifyKindSetTrait =>
+ // it’s satisfied when modify kinds are included in the required set,
+ // e.g. [I,U] satisfy [I,U,D]
+ // [I,U,D] not satisfy [I,D]
+ this.modifyKindSet.getContainedKinds.forall(other.modifyKindSet.contains)
+ case other: UpdateKindTrait =>
Review comment:
You are right. This shouldn't be here. It is used to check whether the given UpdateKindTrait is valid. I move this logic into the validation in `FlinkChangelogModeInferenceProgram`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8447d9c54212a7625a656a7ac3f03ae86e8c9b5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
libenchao commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407077655
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/ModifyKind.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+/**
+ * Lists all kinds of modify operations that happens in a changelog.
Review comment:
nit: happens -> happen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407154553
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
+ node: StreamPhysicalRel,
+ children: List[StreamPhysicalRel],
+ providedTrait: ModifyKindSetTrait,
+ requiredTrait: ModifyKindSetTrait,
+ requestedOwner: String): StreamPhysicalRel = {
+ if (!providedTrait.satisfies(requiredTrait)) {
+ val diff = providedTrait.modifyKindSet.diff(requiredTrait.modifyKindSet)
+ val diffString = diff.getContainedKinds
+ .toList.sorted // for deterministic error message
+ .map(_.toString.toLowerCase)
+ .mkString(" and ")
+ // creates a new node based on the new children, to have a more correct node description
+ // e.g. description of GroupAggregate is based on the ModifyKindSetTrait of children
+ val tempNode = node.copy(node.getTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ val nodeString = tempNode.getRelDetailedDescription
+ throw new TableException(
+ s"$requestedOwner doesn't support consuming $diffString changes " +
+ s"which is produced by node $nodeString")
+ }
+ val newTraitSet = node.getTraitSet.plus(providedTrait)
+ node.copy(newTraitSet, children).asInstanceOf[StreamPhysicalRel]
+ }
+ }
+
+ /**
+ * A visitor which will try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[UpdateKindTrait]]
+ * or returns None if the planner doesn't support to satisfy the required [[UpdateKindTrait]].
+ */
+ private class SatisfyUpdateKindTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[UpdateKindTrait]] from root.
+ *
+ * <p>Each node will first require a UpdateKindTrait to its children.
+ * The required UpdateKindTrait may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>If the node will pass the children's UpdateKindTrait without destroying it,
+ * then return a new node with new inputs and forwarded UpdateKindTrait.
+ *
+ * <p>If the node will destroy the children's UpdateKindTrait, then the node itself
+ * needs to be converted, or a new node should be generated to satisfy the required trait,
+ * such as marking itself not to generate UPDATE_BEFORE,
+ * or generating a new node to filter UPDATE_BEFORE.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required UpdateKindTrait
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or None if required traits cannot be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: UpdateKindTrait): Option[StreamPhysicalRel] = rel match {
+ case sink: StreamExecSink[_] =>
+ val childModifyKindSet = getModifyKindSet(sink.getInput)
+ val sinkRequiredTraits = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(onlyAfterOrNoUpdate(childModifyKindSet), beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: RetractStreamTableSink[_] =>
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ case _: AppendStreamTableSink[_] | _: StreamTableSink[_] =>
+ Seq(UpdateKindTrait.NO_UPDATE)
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ if (ds.needUpdateBefore) {
+ Seq(beforeAfterOrNoUpdate(childModifyKindSet))
+ } else {
+ // support both ONLY_AFTER and BEFORE_AFTER, but prefer ONLY_AFTER
+ Seq(
+ onlyAfterOrNoUpdate(childModifyKindSet),
+ beforeAfterOrNoUpdate(childModifyKindSet))
+ }
+ } else {
+ Seq(UpdateKindTrait.NO_UPDATE)
+ }
+ }
+ val children = sinkRequiredTraits.flatMap(t => visitChildren(sink, t))
+ if (children.isEmpty) {
+ None
+ } else {
+ val sinkTrait = sink.getTraitSet.plus(UpdateKindTrait.NO_UPDATE)
+ Some(sink.copy(sinkTrait, children.head).asInstanceOf[StreamPhysicalRel])
+ }
+
+ case _: StreamExecGroupAggregate | _: StreamExecGroupTableAggregate |
+ _: StreamExecLimit =>
+ // Aggregate, TableAggregate and Limit requires update_before if there are updates
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(rel.getInput(0)))
+ val children = visitChildren(rel, requiredChildTrait)
+ // use requiredTrait as providedTrait, because they should support all kinds of UpdateKind
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case _: StreamExecGroupWindowAggregate | _: StreamExecGroupWindowTableAggregate |
+ _: StreamExecDeduplicate | _: StreamExecTemporalSort | _: StreamExecMatch |
+ _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // WindowAggregate, WindowTableAggregate, Deduplicate, TemporalSort, CEP, OverAggregate
+ // and WindowJoin require no updates in input
+ val children = visitChildren(rel, UpdateKindTrait.NO_UPDATE)
+ replaceChildrenAndTrait(rel, children, requiredTrait)
+
+ case rank: StreamExecRank =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, rank.partitionKey, rank.orderKey)
+ // creates new Rank nodes for every applied RankStrategy
+ val newRankNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newRankNodes, requiredTrait)
+
+ case rank: StreamExecSortLimit =>
+ val rankStrategies = RankProcessStrategy.analyzeRankProcessStrategies(
+ rank.getInput, ImmutableBitSet.of(), rank.getCollation)
+ // creates new SortLimit nodes for every applied RankStrategy
+ val newSortLimitNodes = rankStrategies.map(rank.copy)
+ visitRankStrategies(rankStrategies, newSortLimitNodes, requiredTrait)
+
+ case sort: StreamExecSort =>
+ val requiredChildTrait = beforeAfterOrNoUpdate(getModifyKindSet(sort.getInput))
+ val children = visitChildren(sort, requiredChildTrait)
+ replaceChildrenAndTrait(sort, children, requiredTrait)
+
+ case join: StreamExecJoin =>
+ val requiredUpdateBeforeByParent = requiredTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ val children = List(0, 1).map { childOrdinal =>
+ val child = join.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ val needUpdateBefore = !join.inputUniqueKeyContainsJoinKey(childOrdinal)
+ val inputModifyKindSet = getModifyKindSet(child)
+ val childRequiredTrait = if (needUpdateBefore || requiredUpdateBeforeByParent) {
+ beforeAfterOrNoUpdate(inputModifyKindSet)
+ } else {
+ onlyAfterOrNoUpdate(inputModifyKindSet)
+ }
+ this.visit(child, childRequiredTrait)
+ }
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ replaceChildrenAndTrait(join, Some(children.flatten), requiredTrait)
+ }
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // forward required mode to left input
+ val left = temporalJoin.getLeft.asInstanceOf[StreamPhysicalRel]
+ val right = temporalJoin.getRight.asInstanceOf[StreamPhysicalRel]
+ val newLeftOption = this.visit(left, requiredTrait)
+ // currently temporal join only support insert-only source as the right side
+ val newRightOption = this.visit(right, UpdateKindTrait.NO_UPDATE)
+ (newLeftOption, newRightOption) match {
+ case (Some(newLeft), Some(newRight)) =>
+ val leftTrait = newLeft.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, Some(List(newLeft, newRight)), leftTrait)
+ case _ =>
+ None
+ }
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ visitChildren(rel, requiredTrait) match {
+ case None => None
+ case Some(children) =>
+ val childTrait = children.head.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ replaceChildrenAndTrait(rel, Some(children), childTrait)
+ }
+
+ case union: StreamExecUnion =>
+ val children = union.getInputs.map { case child: StreamPhysicalRel =>
+ val childModifyKindSet = getModifyKindSet(child)
+ val requiredChildTrait = if (childModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ requiredTrait
+ }
+ this.visit(child, requiredChildTrait)
+ }.toList
+ if (children.exists(_.isEmpty)) {
+ None
+ } else {
+ val updateKinds = children.flatten
+ .map(_.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE))
+ // union can just forward changes, can't actively satisfy to another changelog mode
+ val providedTrait = if (updateKinds.forall(k => UpdateKindTrait.NO_UPDATE == k)) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ updateKinds.head
+ }
+ replaceChildrenAndTrait(union, Some(children.flatten), providedTrait)
+
+ }
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ replaceChildrenAndTrait(rel, Some(List()), UpdateKindTrait.NO_UPDATE)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = if (scan.intermediateTable.isUpdateBeforeRequired) {
+ // we can't drop UPDATE_BEFORE if it is required by other parent blocks
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else {
+ requiredTrait
+ }
+ if (!providedTrait.satisfies(requiredTrait)) {
+ // require ONLY_AFTER but can only provide BEFORE_AND_AFTER
+ None
+ } else {
+ replaceChildrenAndTrait(rel, Some(List()), providedTrait)
+ }
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: UpdateKindTrait): Option[List[StreamPhysicalRel]] = {
+ val newChildren = for (child <- parent.getInputs) yield {
+ this.visit(child.asInstanceOf[StreamPhysicalRel], requiredChildrenTrait) match {
+ case None =>
+ // return None if one of the children can't satisfy
+ return None
+ case Some(newChild) =>
+ val providedTrait = newChild.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ val childDescription = newChild.getRelDetailedDescription
+ if (!providedTrait.satisfies(requiredChildrenTrait)) {
+ throw new TableException(s"Provided trait $providedTrait can't satisfy " +
+ s"required trait $requiredChildrenTrait. " +
+ s"This is a bug in planner, please file an issue. \n" +
+ s"Current node is $childDescription")
+ }
+ newChild
+ }
+ }
+ Some(newChildren.toList)
+ }
+
+ private def replaceChildrenAndTrait(
Review comment:
rename to `createNewNode`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407154687
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdModifiedMonotonicity.scala
##########
@@ -41,6 +40,7 @@ import org.apache.calcite.sql.validate.SqlMonotonicity
import org.apache.calcite.sql.validate.SqlMonotonicity._
import org.apache.calcite.sql.{SqlKind, SqlOperatorBinding}
import org.apache.calcite.util.Util
+import org.apache.flink.table.planner.plan.utils.ChangelogPlanUtils
Review comment:
revert this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 6d34187c1dde98f1fc8573063e298bd481224688 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160017412) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384)
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407134113
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/ChangelogPlanUtils.scala
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils
+
+import org.apache.flink.table.connector.ChangelogMode
+import org.apache.flink.table.planner.plan.`trait`.{ModifyKind, ModifyKindSetTraitDef, UpdateKind, UpdateKindTraitDef}
+import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalRel
+import org.apache.flink.table.planner.plan.optimize.program.FlinkChangelogModeInferenceProgram
+import org.apache.flink.types.RowKind
+import org.apache.calcite.rel.RelNode
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Utilities for changelog plan.
+ */
+object ChangelogPlanUtils {
+
+ /**
+ * Returns true if the plan produces insert-only changes.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def isInsertOnly(plan: RelNode): Boolean = {
+ assert(plan.isInstanceOf[StreamPhysicalRel])
+ val modifyKindSetTrait = plan.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ modifyKindSetTrait.modifyKindSet.isInsertOnly
+ }
+
+ /**
+ * Returns true if the [[RelNode]] will generate UPDATE_BEFORE messages.
+ * This method is used to determine whether the runtime operator should
+ * produce UPDATE_BEFORE messages with UPDATE_AFTER message together.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def generateUpdateBefore(node: RelNode): Boolean = {
+ assert(node.isInstanceOf[StreamPhysicalRel])
+ val updateKindTrait = node.getTraitSet.getTrait(UpdateKindTraitDef.INSTANCE)
+ updateKindTrait.updateKind == UpdateKind.BEFORE_AND_AFTER
+ }
+
+ /**
+ * Gets an optional [[ChangelogMode]] of the given physical node.
+ * The [[ChangelogMode]] is inferred from ModifyKindSetTrait and UpdateKindTrait.
+ * The returned value is None if the given node is Sink node.
+ *
+ * <p>Note: this method must be called after [[FlinkChangelogModeInferenceProgram]] is applied.
+ */
+ def getChangelogMode(node: RelNode): Option[ChangelogMode] = {
+ assert(node.isInstanceOf[StreamPhysicalRel])
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* b8ea858f10d83717561e894c71e5b0b49fbd85f4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160141637) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417)
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407145550
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.optimize.program
+
+import org.apache.flink.table.api.TableException
+import org.apache.flink.table.planner.plan.`trait`.UpdateKindTrait.{beforeAfterOrNoUpdate, onlyAfterOrNoUpdate}
+import org.apache.flink.table.planner.plan.`trait`._
+import org.apache.flink.table.planner.plan.nodes.physical.stream._
+import org.apache.flink.table.planner.plan.utils._
+import org.apache.flink.table.planner.sinks.DataStreamTableSink
+import org.apache.flink.table.runtime.operators.join.FlinkJoinType
+import org.apache.flink.table.sinks.{AppendStreamTableSink, RetractStreamTableSink, StreamTableSink, UpsertStreamTableSink}
+
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.util.ImmutableBitSet
+
+import scala.collection.JavaConversions._
+
+/**
+ * An optimize program to infer ChangelogMode for every physical nodes.
+ */
+class FlinkChangelogModeInferenceProgram extends FlinkOptimizeProgram[StreamOptimizeContext] {
+
+ private val SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR = new SatisfyModifyKindSetTraitVisitor
+ private val SATISFY_UPDATE_KIND_TRAIT_VISITOR = new SatisfyUpdateKindTraitVisitor
+
+ override def optimize(
+ root: RelNode,
+ context: StreamOptimizeContext): RelNode = {
+
+ // step1: satisfy ModifyKindSet trait
+ val physicalRoot = root.asInstanceOf[StreamPhysicalRel]
+ val rootWithModifyKindSet = SATISFY_MODIFY_KIND_SET_TRAIT_VISITOR.visit(
+ physicalRoot,
+ // we do not propagate the ModifyKindSet requirement and requester among blocks
+ // set default ModifyKindSet requirement and requester for root
+ ModifyKindSetTrait.ALL_CHANGES,
+ "ROOT")
+
+ // step2: satisfy UpdateKind trait
+ val rootModifyKindSet = getModifyKindSet(rootWithModifyKindSet)
+ // use the required UpdateKindTrait from parent blocks
+ val requiredUpdateKindTrait = if (context.isUpdateBeforeRequired) {
+ UpdateKindTrait.BEFORE_AND_AFTER
+ } else if (rootModifyKindSet.isInsertOnly) {
+ UpdateKindTrait.NO_UPDATE
+ } else {
+ UpdateKindTrait.ONLY_UPDATE_AFTER
+ }
+ val finalRoot = SATISFY_UPDATE_KIND_TRAIT_VISITOR.visit(
+ rootWithModifyKindSet,
+ requiredUpdateKindTrait)
+
+ // step3: sanity check and return non-empty root
+ if (finalRoot.isEmpty) {
+ val plan = FlinkRelOptUtil.toString(root, withChangelogTraits = true)
+ throw new TableException(
+ "Can't generate a valid execution plan for the given query:\n" + plan)
+ } else {
+ finalRoot.get
+ }
+ }
+
+
+ /**
+ * A visitor which will try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>After traversed by this visitor, every node should have a correct [[ModifyKindSetTrait]]
+ * or an exception should be thrown if the planner doesn't support to satisfy the required
+ * [[ModifyKindSetTrait]].
+ */
+ private class SatisfyModifyKindSetTraitVisitor {
+
+ /**
+ * Try to satisfy the required [[ModifyKindSetTrait]] from root.
+ *
+ * <p>Each node should first require a [[ModifyKindSetTrait]] to its children.
+ * If the trait provided by children does not satisfy the required one,
+ * it should throw an exception and prompt the user that plan is not supported.
+ * The required [[ModifyKindSetTrait]] may come from the node's parent,
+ * or come from the node itself, depending on whether the node will destroy
+ * the trait provided by children or pass the trait from children.
+ *
+ * <p>Each node should provide [[ModifyKindSetTrait]] according to current node behavior
+ * and the ModifyKindSetTrait provided by children.
+ *
+ * @param rel the node who should satisfy the requiredTrait
+ * @param requiredTrait the required ModifyKindSetTrait
+ * @param requester the requester who starts the requirement, used for better exception message
+ * @return A converted node which satisfy required traits by inputs node of current node.
+ * Or throws exception if required trait can’t be satisfied.
+ */
+ def visit(
+ rel: StreamPhysicalRel,
+ requiredTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = rel match {
+ case sink: StreamExecSink[_] =>
+ val (sinkRequiredTrait, name) = sink.sink match {
+ case _: UpsertStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "UpsertStreamTableSink")
+ case _: RetractStreamTableSink[_] =>
+ (ModifyKindSetTrait.ALL_CHANGES, "RetractStreamTableSink")
+ case _: AppendStreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "AppendStreamTableSink")
+ case _: StreamTableSink[_] =>
+ (ModifyKindSetTrait.INSERT_ONLY, "StreamTableSink")
+ case ds: DataStreamTableSink[_] =>
+ if (ds.withChangeFlag) {
+ (ModifyKindSetTrait.ALL_CHANGES, "toRetractStream")
+ } else {
+ (ModifyKindSetTrait.INSERT_ONLY, "toAppendStream")
+ }
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported sink '${sink.sink.getClass.getSimpleName}'")
+ }
+ val children = visitChildren(sink, sinkRequiredTrait, name)
+ val sinkTrait = sink.getTraitSet.plus(ModifyKindSetTrait.EMPTY)
+ // ignore required trait from context, because sink is the true root
+ sink.copy(sinkTrait, children).asInstanceOf[StreamPhysicalRel]
+
+ case deduplicate: StreamExecDeduplicate =>
+ // deduplicate only support insert only as input
+ val children = visitChildren(deduplicate, ModifyKindSetTrait.INSERT_ONLY)
+ val providedTrait = if (deduplicate.keepLastRow) {
+ // produce updates if it keeps last row
+ ModifyKindSetTrait.ALL_CHANGES
+ } else {
+ ModifyKindSetTrait.INSERT_ONLY
+ }
+ replaceChildrenAndTrait(deduplicate, children, providedTrait, requiredTrait, requester)
+
+ case agg: StreamExecGroupAggregate =>
+ // agg support all changes in input
+ val children = visitChildren(agg, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet = getModifyKindSet(children.head)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ .addContainedKind(ModifyKind.UPDATE)
+ if (inputModifyKindSet.contains(ModifyKind.UPDATE) ||
+ inputModifyKindSet.contains(ModifyKind.DELETE)) {
+ builder.addContainedKind(ModifyKind.DELETE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(agg, children, providedTrait, requiredTrait, requester)
+
+ case tagg: StreamExecGroupTableAggregate =>
+ // table agg support all changes in input
+ val children = visitChildren(tagg, ModifyKindSetTrait.ALL_CHANGES)
+ // table aggregate will produce all changes, including deletions
+ replaceChildrenAndTrait(
+ tagg, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case window: StreamExecGroupWindowAggregateBase =>
+ // WindowAggregate and WindowTableAggregate support insert-only in input
+ val children = visitChildren(window, ModifyKindSetTrait.INSERT_ONLY)
+ val builder = ModifyKindSet.newBuilder()
+ .addContainedKind(ModifyKind.INSERT)
+ if (window.emitStrategy.produceUpdates) {
+ builder.addContainedKind(ModifyKind.UPDATE)
+ }
+ val providedTrait = new ModifyKindSetTrait(builder.build())
+ replaceChildrenAndTrait(window, children, providedTrait, requiredTrait, requester)
+
+ case limit: StreamExecLimit =>
+ // limit support all changes in input
+ val children = visitChildren(limit, ModifyKindSetTrait.ALL_CHANGES)
+ val providedTrait = if (getModifyKindSet(children.head).isInsertOnly) {
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(limit, children, providedTrait, requiredTrait, requester)
+
+ case _: StreamExecRank | _: StreamExecSortLimit =>
+ // Rank supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.ALL_CHANGES, requiredTrait, requester)
+
+ case sort: StreamExecSort =>
+ // Sort supports consuming all changes
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ // Sort will buffer all inputs, and produce insert-only messages when input is finished
+ replaceChildrenAndTrait(
+ sort, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case cep: StreamExecMatch =>
+ // CEP only supports consuming insert-only and producing insert-only changes
+ // give a better requester name for exception message
+ val children = visitChildren(cep, ModifyKindSetTrait.INSERT_ONLY, "Match Recognize")
+ replaceChildrenAndTrait(
+ cep, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case _: StreamExecTemporalSort | _: StreamExecOverAggregate | _: StreamExecWindowJoin =>
+ // TemporalSort, OverAggregate, WindowJoin only support consuming insert-only
+ // and producing insert-only changes
+ val children = visitChildren(rel, ModifyKindSetTrait.INSERT_ONLY)
+ replaceChildrenAndTrait(
+ rel, children, ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case join: StreamExecJoin =>
+ // join support all changes in input
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ val innerOrSemi = join.flinkJoinType == FlinkJoinType.INNER ||
+ join.flinkJoinType == FlinkJoinType.SEMI
+ val providedTrait = if (leftKindSet.isInsertOnly &&
+ rightKindSet.isInsertOnly && innerOrSemi) {
+ // produce insert-only because results are deterministic
+ ModifyKindSetTrait.INSERT_ONLY
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ replaceChildrenAndTrait(join, children, providedTrait, requiredTrait, requester)
+
+ case temporalJoin: StreamExecTemporalJoin =>
+ // currently, temporal join only support insert-only input streams, including right side
+ val children = visitChildren(temporalJoin, ModifyKindSetTrait.INSERT_ONLY)
+ // forward left input changes
+ val leftTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ replaceChildrenAndTrait(temporalJoin, children, leftTrait, requiredTrait, requester)
+
+ case _: StreamExecCalc | _: StreamExecPythonCalc | _: StreamExecCorrelate |
+ _: StreamExecPythonCorrelate | _: StreamExecLookupJoin | _: StreamExecExchange |
+ _: StreamExecExpand | _: StreamExecMiniBatchAssigner |
+ _: StreamExecWatermarkAssigner =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ val childrenTrait = children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
+ // forward children mode
+ replaceChildrenAndTrait(rel, children, childrenTrait, requiredTrait, requester)
+
+ case union: StreamExecUnion =>
+ // transparent forward requiredTrait to children
+ val children = visitChildren(rel, requiredTrait, requester)
+ // union provides all possible kinds of children have
+ val providedKindSet = ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ replaceChildrenAndTrait(
+ union, children, new ModifyKindSetTrait(providedKindSet), requiredTrait, requester)
+
+ case _: StreamExecDataStreamScan | _: StreamExecTableSourceScan | _: StreamExecValues =>
+ // DataStream, TableSource and Values only support producing insert-only messages
+ replaceChildrenAndTrait(
+ rel, List(), ModifyKindSetTrait.INSERT_ONLY, requiredTrait, requester)
+
+ case scan: StreamExecIntermediateTableScan =>
+ val providedTrait = new ModifyKindSetTrait(scan.intermediateTable.modifyKindSet)
+ replaceChildrenAndTrait(scan, List(), providedTrait, requiredTrait, requester)
+
+ case _ =>
+ throw new UnsupportedOperationException(
+ s"Unsupported visit for ${rel.getClass.getSimpleName}")
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait): List[StreamPhysicalRel] = {
+ visitChildren(parent, requiredChildrenTrait, getNodeName(parent))
+ }
+
+ private def visitChildren(
+ parent: StreamPhysicalRel,
+ requiredChildrenTrait: ModifyKindSetTrait,
+ requester: String): List[StreamPhysicalRel] = {
+ val newChildren = for (i <- 0 until parent.getInputs.size()) yield {
+ visitChild(parent, i, requiredChildrenTrait, requester)
+ }
+ newChildren.toList
+ }
+
+ private def visitChild(
+ parent: StreamPhysicalRel,
+ childOrdinal: Int,
+ requiredChildTrait: ModifyKindSetTrait,
+ requester: String): StreamPhysicalRel = {
+ val child = parent.getInput(childOrdinal).asInstanceOf[StreamPhysicalRel]
+ this.visit(child, requiredChildTrait, requester)
+ }
+
+ private def getNodeName(rel: StreamPhysicalRel): String = {
+ val prefix = "StreamExec"
+ val typeName = rel.getRelTypeName
+ if (typeName.startsWith(prefix)) {
+ typeName.substring(prefix.length)
+ } else {
+ typeName
+ }
+ }
+
+ private def replaceChildrenAndTrait(
Review comment:
rename this method to `createNewNode`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407438410
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/trait/UpdateKind.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.trait;
+
+/**
+ * Lists all kinds of {@link ModifyKind#UPDATE} operation.
+ */
+public enum UpdateKind {
+
+ /**
+ * No update operation.
+ */
+ NO_UPDATE,
+
+ /**
+ * This kind indicates that operators should emit update changes just as a row of
+ * {@code RowKind#UPDATE_AFTER}.
+ */
+ ONLY_UPDATE_AFTER,
+
+ /**
+ * This kind indicates that operators should emit update changes in the way that
+ * a row of {@code RowKind#UPDATE_BEFORE} and a row of {@code RowKind#UPDATE_AFTER} together.
+ */
+ BEFORE_AND_AFTER;
+
+ /**
+ * Return a new UpdateKind which merges this UpdateKind and the other UpdateKind.
+ */
+ public UpdateKind merge(UpdateKind that) {
Review comment:
Children of Union may have different UpdateKind. This is used to "union" children's UpdateKind. I was thinking "union" is not fit for this enum class, because it is not a bitset.
I moved this logic into `FlinkChangelogModeInferenceProgram` as it is only used by StreamExecUnion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "732829017006414abc142159ff2d819b57fe34c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "732829017006414abc142159ff2d819b57fe34c7",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160141637",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7417",
"triggerID" : "b8ea858f10d83717561e894c71e5b0b49fbd85f4",
"triggerType" : "PUSH"
}, {
"hash" : "8a496f496bbe955bd33460443eb53a696f612a68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a496f496bbe955bd33460443eb53a696f612a68",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160237416",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
}, {
"hash" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473",
"triggerID" : "b33d8e0e64c7b8302d2f495340d8399b88c3b543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 732829017006414abc142159ff2d819b57fe34c7 UNKNOWN
* 8a496f496bbe955bd33460443eb53a696f612a68 UNKNOWN
* b33d8e0e64c7b8302d2f495340d8399b88c3b543 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160237416) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7473)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#issuecomment-611092822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7236",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159412901",
"triggerID" : "d8447d9c54212a7625a656a7ac3f03ae86e8c9b5",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7248",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159519229",
"triggerID" : "6813ce29fd9023e8cba34f9539ed09b080401ba1",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159526393",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7253",
"triggerID" : "59ab8cedf26f3b55602b448951fa18e8e7997bb5",
"triggerType" : "PUSH"
}, {
"hash" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27728b10f3e719b3f53fe199cf02e5b820ddfcd1",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160017412",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "6d34187c1dde98f1fc8573063e298bd481224688",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7384",
"triggerID" : "6d34187c1dde98f1fc8573063e298bd481224688",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160023748",
"triggerID" : "32424f2a02f4f3aff2d32f3076db6841e2dec32d",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160066338",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409",
"triggerID" : "3e2f82101f69468111f5b3d1d1d380e6546036c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27728b10f3e719b3f53fe199cf02e5b820ddfcd1 UNKNOWN
* 32424f2a02f4f3aff2d32f3076db6841e2dec32d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160023748) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7388)
* 3e2f82101f69468111f5b3d1d1d380e6546036c7 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160066338) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7409)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407197919
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/physical/stream/TwoStageOptimizedAggregateRule.scala
##########
@@ -131,7 +130,9 @@ class TwoStageOptimizedAggregateRule extends RelOptRule(
input.getCluster.getTypeFactory.asInstanceOf[FlinkTypeFactory])
// local agg shouldn't produce AccRetract Message
Review comment:
update the comment too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #11674:
[FLINK-16887][table-planner-blink] Refactor retraction rules to support
inferring ChangelogMode
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #11674: [FLINK-16887][table-planner-blink] Refactor retraction rules to support inferring ChangelogMode
URL: https://github.com/apache/flink/pull/11674#discussion_r407135562
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/trait/UpdateKindTraitDef.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.`trait`
+
+import org.apache.calcite.plan.{RelOptPlanner, RelTraitDef}
+import org.apache.calcite.rel.RelNode
+
+class UpdateKindTraitDef extends RelTraitDef[UpdateKindTrait] {
+
+ override def getTraitClass: Class[UpdateKindTrait] = classOf[UpdateKindTrait]
+
+ override def getSimpleName: String = this.getClass.getSimpleName
+
+ override def convert(
+ planner: RelOptPlanner,
+ rel: RelNode,
+ toTrait: UpdateKindTrait,
+ allowInfiniteCostConverters: Boolean): RelNode = {
+ rel.copy(rel.getTraitSet.plus(toTrait), rel.getInputs)
+ }
+
+ override def canConvert(
+ planner: RelOptPlanner,
+ fromTrait: UpdateKindTrait,
+ toTrait: UpdateKindTrait): Boolean = true
Review comment:
we should throw exception here which means `UpdateKindTraitDef ` does not support volcano planner now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services