You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by ChengXiangLi <gi...@git.apache.org> on 2016/01/28 12:53:05 UTC
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
GitHub user ChengXiangLi opened a pull request:
https://github.com/apache/flink/pull/1556
[Flink-3226] Implement project and filter translator.
This is just prototype implementation to translate Calcite Project and Filter into Flink RelNode. It still miss several parts, which could be done in separate tasks:
1. expression code generation.
2. Flink expression do not support `eval` yet, so expression is not supported in interpreter mode as well.
3. i test these rules based on Timo's prototype, the rule registry and launch part is not ready in this PR.
4. other Calcite RelNodes translator.
@fhueske and @twalthr , please let me know if you have any suggestion.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ChengXiangLi/flink flink-3226
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1556.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1556
----
commit 963dceee1e0e4456f07f7150f218ee23a34719be
Author: chengxiang li <ch...@intel.com>
Date: 2016-01-28T11:04:08Z
Implement project and filter translator.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/1556#discussion_r51125883
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/api/table/sql/calcite/flinkFunction/ProjectFunction.java ---
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.sql.calcite.flinkFunction;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.table.Row;
+import org.apache.flink.api.table.expressions.Expression;
+import org.apache.flink.api.table.expressions.Naming;
+import org.apache.flink.api.table.typeinfo.RowTypeInfo;
+
+import java.util.List;
+
+public class ProjectFunction extends RichMapFunction<Row, Row> {
+
+ private RowTypeInfo outputType;
+ private List<Naming> expressions;
+
+ public ProjectFunction(RowTypeInfo outputType, List<Naming> expressions) {
--- End diff --
We already had the discussion that TypeInfo should not be shipped to the cluster. We need to generate the `Function`s too, so that map() will work on the indexes of the row directly without name information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the pull request:
https://github.com/apache/flink/pull/1556#issuecomment-177517981
@ChengXiangLi The code generation will be similar to the old Table API. The main difference is that the code generation already happens at the client side. The generated code is passed to a wrapper `Function`. The compilation happens on the cluster but TypeInformation is not shipped there anymore. I will try to come up with an example of a project operator tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by ChengXiangLi <gi...@git.apache.org>.
Github user ChengXiangLi commented on the pull request:
https://github.com/apache/flink/pull/1556#issuecomment-177541447
I got it, as most of the work is done by code generation, current DataSetProjectRule/DataSetFilterRule/DataSetCalRule implementation should be ready. Most part of `DataSetJoinRule` and `DataSetAggregateRule` has been finished from my side, i would open a new PR tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by ChengXiangLi <gi...@git.apache.org>.
Github user ChengXiangLi commented on the pull request:
https://github.com/apache/flink/pull/1556#issuecomment-176894471
Thanks for the comments, i would create a new PR based on Fabian's work. Besides, @twalthr , what's the scope of your code generation, previous Table API code generation generate MapFunction/FilterFunction directly, if you plan to do the same thing, i think DataSetProjectRule/DataSetFilterRule/DataSetCalRule in Fabian's PR should be ready yet. If not, please let me know your interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the pull request:
https://github.com/apache/flink/pull/1556#issuecomment-176157407
As far as I know the `Expressions` are for the user-facing API only, right? I think we won't need the `RexToExpr` class anymore, as I am currently writing a rex to code generator.
You have implemented `Function` with row types by default. In the old Table API we also generated these functions in order to not only support row types. In the future we could also use Tuple type where possible to improve the performance. Do we want lose this possibility?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3226] Implement project and filter tran...
Posted by ChengXiangLi <gi...@git.apache.org>.
Github user ChengXiangLi closed the pull request at:
https://github.com/apache/flink/pull/1556
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---