You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by twalthr <gi...@git.apache.org> on 2017/01/05 13:26:58 UTC
[GitHub] flink pull request #3063: [FLINK-5357] [table] Fix dropped projections
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/3063
[FLINK-5357] [table] Fix dropped projections
In certain cases Calcite drops projections which is not what we want (e.g. using WordCount example). This PR adds explicit projections which are removed by optimizer rules later.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink FLINK-5357
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3063.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3063
----
commit 0966b8f1fc60ecebeb0a36817c4f2dcb5f1547f6
Author: twalthr <tw...@apache.org>
Date: 2017-01-05T13:14:52Z
[FLINK-5357] [table] Fix dropped projections
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3063: [FLINK-5357] [table] Fix dropped projections
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3063
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3063: [FLINK-5357] [table] Fix dropped projections
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/3063
Thanks for the review @KurtYoung and @wuchong.
I think we should use Calcite's RelBuilder whenever possible. In fact the current solution was a workaround because RelBuilder would drop identity projections that renamed attributes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3063: [FLINK-5357] [table] Fix dropped projections
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/3063#discussion_r95308316
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/logical/operators.scala ---
@@ -92,20 +92,19 @@ case class Project(projectList: Seq[NamedExpression], child: LogicalNode) extend
}
override protected[logical] def construct(relBuilder: RelBuilder): RelBuilder = {
- val allAlias = projectList.forall(_.isInstanceOf[Alias])
child.construct(relBuilder)
- if (allAlias) {
- // Calcite's RelBuilder does not translate identity projects even if they rename fields.
- // Add a projection ourselves (will be automatically removed by translation rules).
- val project = LogicalProject.create(relBuilder.peek(),
+ // Calcite's RelBuilder does not translate identity projects even if they rename fields.
+ // We add a projection ourselves (will be automatically removed by translation rules).
+ val project = LogicalProject.create(
--- End diff --
Good point ! The hack code existed because of Calcite's RelBuilder does not create project when only rename fields. The `project( Iterable<? extends RexNode> nodes, Iterable<String> fieldNames, boolean force)` was introduced and fixed this issue by Calcite-1342.
I have tried to replace the hack code by the following code, and all tests passed !
```scala
override protected[logical] def construct(relBuilder: RelBuilder): RelBuilder = {
child.construct(relBuilder)
relBuilder.project(
projectList.map {
case Alias(c, _, _) => c.toRexNode(relBuilder)
case n@_ => n.toRexNode(relBuilder)
}.asJava,
projectList.map(_.name).asJava,
true)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3063: [FLINK-5357] [table] Fix dropped projections
Posted by KurtYoung <gi...@git.apache.org>.
Github user KurtYoung commented on a diff in the pull request:
https://github.com/apache/flink/pull/3063#discussion_r95304042
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/logical/operators.scala ---
@@ -92,20 +92,19 @@ case class Project(projectList: Seq[NamedExpression], child: LogicalNode) extend
}
override protected[logical] def construct(relBuilder: RelBuilder): RelBuilder = {
- val allAlias = projectList.forall(_.isInstanceOf[Alias])
child.construct(relBuilder)
- if (allAlias) {
- // Calcite's RelBuilder does not translate identity projects even if they rename fields.
- // Add a projection ourselves (will be automatically removed by translation rules).
- val project = LogicalProject.create(relBuilder.peek(),
+ // Calcite's RelBuilder does not translate identity projects even if they rename fields.
+ // We add a projection ourselves (will be automatically removed by translation rules).
+ val project = LogicalProject.create(
--- End diff --
I noticed that Calcite's RelBuilder has this method:
`project(
Iterable<? extends RexNode> nodes,
Iterable<String> fieldNames,
boolean force)`
Would it be more consistent to use this method by setting the force to `true`, instead of creating Projection by ourself.
But either is fine with me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---