You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by godfreyhe <gi...@git.apache.org> on 2017/01/12 11:16:18 UTC

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

GitHub user godfreyhe opened a pull request:

    https://github.com/apache/flink/pull/3101

    [FLINK-5406] [table] add normalization phase for predicate logical plan rewriting

    Normalization phase is for predicate logical plan rewriting and is independent of cost module. The rules in normalization phase do not need to repeatedly applied to different logical plan which is different to volcano optimization phase. And the benefit of normalization phase is to reduce the running time of volcano planner.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/godfreyhe/flink master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3101.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3101
    
----
commit c9f11e7c38f921a403b56a5c124974eb66bddcac
Author: godfreyhe <go...@163.com>
Date:   2017-01-12T10:42:49Z

    add normalization phase for predicate logical plan rewriting

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97992130
  
    --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/calcite/RuleSetConfig.scala ---
    @@ -0,0 +1,140 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.table.calcite
    +
    +import org.apache.calcite.plan.RelOptRule
    +import org.apache.calcite.tools.{RuleSet, RuleSets}
    +import org.apache.flink.util.Preconditions
    +
    +import scala.collection.JavaConverters._
    +
    +/**
    +  * Builder for creating a RuleSet configuration.
    +  */
    +class RuleSetConfigBuilder {
    --- End diff --
    
    I think we don't need a builder in a builder. Can we integrate `RuleSetConfigBuilder` in `CalciteConfigBuilder`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97996845
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/NormalizationRulesTests.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.table
    +
    +import org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule
    +import org.apache.calcite.tools.RuleSets
    +import org.apache.flink.api.scala._
    +import org.apache.flink.table.api.scala._
    +import org.apache.flink.table.calcite.{CalciteConfig, CalciteConfigBuilder, RuleSetConfig,
    +RuleSetConfigBuilder}
    +import org.apache.flink.table.utils.TableTestBase
    +import org.apache.flink.table.utils.TableTestUtil._
    +import org.junit.Test
    +
    +class NormalizationRulesTests extends TableTestBase {
    +
    +  @Test
    +  def testAppleNormalizationRuleForForBatchSQL(): Unit = {
    +    val util = batchTestUtil()
    +    val rc: RuleSetConfig = new RuleSetConfigBuilder()
    +      // rewrite distinct aggregate
    +      .replaceNormRuleSet(RuleSets.ofList(AggregateExpandDistinctAggregatesRule.JOIN))
    +      .replaceOptRuleSet(RuleSets.ofList())
    +      .build
    --- End diff --
    
    Add parentheses to prevent strict IDE warnings.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97993265
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/NormalizationRulesTests.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.table
    +
    +import org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule
    +import org.apache.calcite.tools.RuleSets
    +import org.apache.flink.api.scala._
    +import org.apache.flink.table.api.scala._
    +import org.apache.flink.table.calcite.{CalciteConfig, CalciteConfigBuilder, RuleSetConfig,
    +RuleSetConfigBuilder}
    +import org.apache.flink.table.utils.TableTestBase
    +import org.apache.flink.table.utils.TableTestUtil._
    +import org.junit.Test
    +
    +class NormalizationRulesTests extends TableTestBase {
    --- End diff --
    
    Merge this class with `CalciteConfigBuilderTest`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97997077
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/NormalizationRulesTests.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.table
    +
    +import org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule
    +import org.apache.calcite.tools.RuleSets
    +import org.apache.flink.api.scala._
    +import org.apache.flink.table.api.scala._
    +import org.apache.flink.table.calcite.{CalciteConfig, CalciteConfigBuilder, RuleSetConfig,
    +RuleSetConfigBuilder}
    +import org.apache.flink.table.utils.TableTestBase
    +import org.apache.flink.table.utils.TableTestUtil._
    +import org.junit.Test
    +
    +class NormalizationRulesTests extends TableTestBase {
    +
    +  @Test
    +  def testAppleNormalizationRuleForForBatchSQL(): Unit = {
    --- End diff --
    
    `Apply`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97992445
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/AggregationTest.scala ---
    @@ -165,14 +165,20 @@ class AggregationTest extends TableTestBase {
         val util = batchTestUtil()
         val sourceTable = util.addTable[(Int, Long, Int)]("MyTable", 'a, 'b, 'c)
     
    -    val resultTable = sourceTable.groupBy('a)
    +    // Move "where" before "groupBy" for the former query would generate
    +    // nondeterministic plans with same cost. If we change FlinkRuleSets.DATASET_OPT_RULES,
    +    // the importance of relNode may change, and the test may fail. This issue is mentioned
    +    // in FLINK-5394, we could move "where" to the end when FLINK-5394 is fixed.
    +    val resultTable = sourceTable.where('a === 1)
    --- End diff --
    
    FLINK-5394 is fixed. So we can remove the changes here right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3101: [FLINK-5406] [table] add normalization phase for predicat...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:

    https://github.com/apache/flink/pull/3101
  
    I will shepherd this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3101: [FLINK-5406] [table] add normalization phase for predicat...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:

    https://github.com/apache/flink/pull/3101
  
    Thanks for the update @godfreyhe!
    PR looks good to me and can be merged IMO.
    
    @godfreyhe, can you rebase the PR to the current master?
    @twalthr, do you want to have another look as well?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by godfreyhe <gi...@git.apache.org>.
Github user godfreyhe commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r99524112
  
    --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/calcite/RuleSetConfig.scala ---
    @@ -0,0 +1,140 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.table.calcite
    +
    +import org.apache.calcite.plan.RelOptRule
    +import org.apache.calcite.tools.{RuleSet, RuleSets}
    +import org.apache.flink.util.Preconditions
    +
    +import scala.collection.JavaConverters._
    +
    +/**
    +  * Builder for creating a RuleSet configuration.
    +  */
    +class RuleSetConfigBuilder {
    --- End diff --
    
    If we integrate` RuleSetConfigBuilder` in `CalciteConfigBuilder`, It's better we integrate `RuleSetConfig` in `CalciteConfig` too ? If we want to add another kind of `RuleSet` in future, both `RuleSetConfig` and `CalciteConfigBuilder`  are need to change. Previously, I want to keep `CalciteConfig` and `CalciteConfigBuilder` stable. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by godfreyhe <gi...@git.apache.org>.
Github user godfreyhe commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r99522172
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/AggregationTest.scala ---
    @@ -165,14 +165,20 @@ class AggregationTest extends TableTestBase {
         val util = batchTestUtil()
         val sourceTable = util.addTable[(Int, Long, Int)]("MyTable", 'a, 'b, 'c)
     
    -    val resultTable = sourceTable.groupBy('a)
    +    // Move "where" before "groupBy" for the former query would generate
    +    // nondeterministic plans with same cost. If we change FlinkRuleSets.DATASET_OPT_RULES,
    +    // the importance of relNode may change, and the test may fail. This issue is mentioned
    +    // in FLINK-5394, we could move "where" to the end when FLINK-5394 is fixed.
    +    val resultTable = sourceTable.where('a === 1)
    --- End diff --
    
    OK


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3101: [FLINK-5406] [table] add normalization phase for predicat...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:

    https://github.com/apache/flink/pull/3101
  
    Thanks for the update @godfreyhe.
    I will go over the code a last time and merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3101: [FLINK-5406] [table] add normalization phase for predicat...

Posted by godfreyhe <gi...@git.apache.org>.
Github user godfreyhe commented on the issue:

    https://github.com/apache/flink/pull/3101
  
    Done, thanks for the suggest @fhueske


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/3101


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r97996946
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/NormalizationRulesTests.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.table
    +
    +import org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule
    +import org.apache.calcite.tools.RuleSets
    +import org.apache.flink.api.scala._
    +import org.apache.flink.table.api.scala._
    +import org.apache.flink.table.calcite.{CalciteConfig, CalciteConfigBuilder, RuleSetConfig,
    +RuleSetConfigBuilder}
    +import org.apache.flink.table.utils.TableTestBase
    +import org.apache.flink.table.utils.TableTestUtil._
    +import org.junit.Test
    +
    +class NormalizationRulesTests extends TableTestBase {
    +
    +  @Test
    +  def testAppleNormalizationRuleForForBatchSQL(): Unit = {
    +    val util = batchTestUtil()
    +    val rc: RuleSetConfig = new RuleSetConfigBuilder()
    +      // rewrite distinct aggregate
    +      .replaceNormRuleSet(RuleSets.ofList(AggregateExpandDistinctAggregatesRule.JOIN))
    +      .replaceOptRuleSet(RuleSets.ofList())
    +      .build
    +
    +    val cc: CalciteConfig = new CalciteConfigBuilder().replaceRuleSetConfig(rc).build
    +    util.tEnv.getConfig.setCalciteConfig(cc)
    +
    +    util.addTable[(Int, Long, String)]("MyTable", 'a, 'b, 'c)
    +
    +    val sqlQuery = "SELECT " +
    +      "COUNT(DISTINCT a)" +
    +      "FROM MyTable group by b"
    +
    +    // expect double aggregate
    +    val expected = unaryNode(
    +      "LogicalProject",
    +      unaryNode("LogicalAggregate",
    +                unaryNode("LogicalAggregate",
    --- End diff --
    
    We use two spaces for indention.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3101: [FLINK-5406] [table] add normalization phase for p...

Posted by godfreyhe <gi...@git.apache.org>.
Github user godfreyhe commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3101#discussion_r100488613
  
    --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/NormalizationRulesTests.scala ---
    @@ -0,0 +1,102 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.table
    +
    +import org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule
    +import org.apache.calcite.tools.RuleSets
    +import org.apache.flink.api.scala._
    +import org.apache.flink.table.api.scala._
    +import org.apache.flink.table.calcite.{CalciteConfig, CalciteConfigBuilder, RuleSetConfig,
    +RuleSetConfigBuilder}
    +import org.apache.flink.table.utils.TableTestBase
    +import org.apache.flink.table.utils.TableTestUtil._
    +import org.junit.Test
    +
    +class NormalizationRulesTests extends TableTestBase {
    --- End diff --
    
    CalciteConfigBuilderTest just tests the correctness of CalciteConfig, does not test logical plan changing when applied a rule. However NormalizationRulesTest tests  the correctness of a logical plan when applied a normalization rule. I suggest we keep NormalizationRulesTest.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---