You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by dawidwys <gi...@git.apache.org> on 2016/04/22 20:11:21 UTC
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
GitHub user dawidwys opened a pull request:
https://github.com/apache/flink/pull/1926
[FLINK-2946] Add orderBy() to Table API
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [ ] General
- The pull request references the related JIRA issue
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dawidwys/flink tableSort
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1926.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1926
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60926272
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.operators.{DataSource, Operator, PartitionOperator}
+import org.apache.flink.api.table.{BatchTableEnvironment, TableConfig}
+import org.apache.flink.api.table.typeutils.TypeConverter
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+
--- End diff --
Please implement `toString` and `explainTerms` similar to the other `DataSetRel`s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60928598
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
--- End diff --
I would use more than three records to test the sorting. Can you use the `CollectionDataSets.get3TupleDataSet()` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60928086
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.operators.{DataSource, Operator, PartitionOperator}
+import org.apache.flink.api.table.{BatchTableEnvironment, TableConfig}
+import org.apache.flink.api.table.typeutils.TypeConverter
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+
+ override def copy(traitSet: RelTraitSet, inputs: util.List[RelNode]): RelNode ={
+ new DataSetSort(
+ cluster,
+ traitSet,
+ inputs.get(0),
+ collations,
+ rowType
+ )
+ }
+
+ override def translateToPlan(
+ tableEnv: BatchTableEnvironment,
+ expectedType: Option[TypeInformation[Any]] = None): DataSet[Any] = {
--- End diff --
We are ignoring the `expectedType`. The convention here is to return a `DataSet` of that type.
If the input type (`inputDS.getType()`) is not equal to the expected type, we need to add a Map function after the sort, which translates the records into the expected type. See `DataSetSource` for an example of how to do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60928727
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
--- End diff --
Can you set the parallelism of the environment to 4 to ensure that the tests are executed in parallel?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61349224
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/package.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.plan.nodes
+
+import org.apache.flink.api.common.functions.MapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.table.TableConfig
+import org.apache.flink.api.table.codegen.CodeGenerator
+import org.apache.flink.api.table.runtime.MapRunner
+
+package object dataset {
+
+ private[dataset] def getConversionMapper(config: TableConfig,
+ inputType: TypeInformation[Any],
--- End diff --
We follow a different indention style and move also the first parameter to the next line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60936652
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.desc)
+
+ val expected = "3,Third\n2,Second\n1,First"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByAsc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.asc)
+
+ val expected = "1,First\n2,Second\n3,Third"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByMultipleFieldsDifferentDirections(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ env.setParallelism(2)
+ val t = env.fromElements((1, 3, "Third"), (1, 2, "Fourth"), (1, 4, "Second"),
+ (2, 1, "Sixth"), (1, 5, "First"), (1, 1, "Fifth"))
+ .toTable(tEnv).orderBy('_1.asc, '_2.desc)
+
+ val expected = "1,5,First\n1,4,Second\n1,3,Third\n1,2,Fourth\n1,1,Fifth\n2,1,Sixth"
+ val results = t.toDataSet[Row].mapPartition(rows => Seq(rows.toSeq)).collect()
+
+ implicit def rowOrdering = Ordering.by((x : Row) => (x.productElement(0).asInstanceOf[Int],
--- End diff --
Not sure about this comment. I think it would(in fact I checked it)... Maybe you missed the next line(81)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60925922
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/table.scala ---
@@ -401,6 +400,52 @@ class Table(
new Table(relBuilder.build(), tableEnv)
}
+ /**
+ * Sorts the given [[Table]]. Similar to SQL ORDER BY. The sorting is performed locally across
--- End diff --
I think the following sentence can be confusing
> The sorting is performed locally across all partitions with keys equal to the given fields.
How about
> The resulting Table is sorted globally sorted across all parallel partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60930292
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.desc)
+
+ val expected = "3,Third\n2,Second\n1,First"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByAsc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.asc)
+
+ val expected = "1,First\n2,Second\n3,Third"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByMultipleFieldsDifferentDirections(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ env.setParallelism(2)
+ val t = env.fromElements((1, 3, "Third"), (1, 2, "Fourth"), (1, 4, "Second"),
+ (2, 1, "Sixth"), (1, 5, "First"), (1, 1, "Fifth"))
+ .toTable(tEnv).orderBy('_1.asc, '_2.desc)
+
+ val expected = "1,5,First\n1,4,Second\n1,3,Third\n1,2,Fourth\n1,1,Fifth\n2,1,Sixth"
+ val results = t.toDataSet[Row].mapPartition(rows => Seq(rows.toSeq)).collect()
+
+ implicit def rowOrdering = Ordering.by((x : Row) => (x.productElement(0).asInstanceOf[Int],
+ - x.productElement(1).asInstanceOf[Int]))
+
+ val result = results.sortBy(p => p.min).reduceLeft(_ ++ _)
--- End diff --
The general method to check for the globally sorted result looks good. Can you do the check for the other tests in a similar way? Setting the sort operator to `1` does not really test the functionality, IMO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-214903475
Hi @dawidwys, thanks for the fast update. I'll have a look at the PR tomorrow.
The JIRA for Table API outer joins is FLINK-2971. I'll assign it to you. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61349399
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/package.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.plan.nodes
+
+import org.apache.flink.api.common.functions.MapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.table.TableConfig
+import org.apache.flink.api.table.codegen.CodeGenerator
+import org.apache.flink.api.table.runtime.MapRunner
+
+package object dataset {
+
+ private[dataset] def getConversionMapper(config: TableConfig,
+ inputType: TypeInformation[Any],
+ expectedType: TypeInformation[Any],
+ conversionOperatorName: String,
+ fieldNames: Seq[String],
+ inputPojoFieldMapping: Option[Array[Int]] = None)
+ : MapFunction[Any, Any] = {
+
+ val generator = new CodeGenerator(config,
--- End diff --
move first parameter to next line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61348408
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/expressions/ordering.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.expressions
+import org.apache.calcite.rex.RexNode
+import org.apache.calcite.tools.RelBuilder
+
+abstract class Ordering extends UnaryExpression { self: Product =>
+}
+
+case class Asc(child: Expression) extends Ordering{
--- End diff --
`Ordering{` -> `Ordering {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61348923
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/package.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.plan.nodes
+
+import org.apache.flink.api.common.functions.MapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.table.TableConfig
+import org.apache.flink.api.table.codegen.CodeGenerator
+import org.apache.flink.api.table.runtime.MapRunner
+
+package object dataset {
+
+ private[dataset] def getConversionMapper(config: TableConfig,
--- End diff --
Can you move this method to `DataSetRel`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60929903
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.desc)
+
+ val expected = "3,Third\n2,Second\n1,First"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByAsc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.asc)
+
+ val expected = "1,First\n2,Second\n3,Third"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByMultipleFieldsDifferentDirections(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ env.setParallelism(2)
+ val t = env.fromElements((1, 3, "Third"), (1, 2, "Fourth"), (1, 4, "Second"),
+ (2, 1, "Sixth"), (1, 5, "First"), (1, 1, "Fifth"))
+ .toTable(tEnv).orderBy('_1.asc, '_2.desc)
+
+ val expected = "1,5,First\n1,4,Second\n1,3,Third\n1,2,Fourth\n1,1,Fifth\n2,1,Sixth"
+ val results = t.toDataSet[Row].mapPartition(rows => Seq(rows.toSeq)).collect()
+
+ implicit def rowOrdering = Ordering.by((x : Row) => (x.productElement(0).asInstanceOf[Int],
--- End diff --
Can you change the `Ordering` such that is does not rely on the actual data (a value (2, 20, "Seventh") would not be sorted correctly).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60926030
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/table.scala ---
@@ -401,6 +400,52 @@ class Table(
new Table(relBuilder.build(), tableEnv)
}
+ /**
+ * Sorts the given [[Table]]. Similar to SQL ORDER BY. The sorting is performed locally across
+ * all partitions with keys equal to the given fields.
+ *
+ * Example:
+ *
+ * {{{
+ * tab.orderBy('name)
--- End diff --
Add `desc` to the example to show how it is used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-213575798
Always glad to help! In fact I would be happy for any suggestions what can I work on further.
I changed the tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-215418083
Thanks for the fast update @dawidwys.
PR is good to merge :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-213565509
Thanks for opening this PR @dawidwys!
I skimmed over the changes and it looks good. I will do a more detailed review hopefully soon.
Thanks also for adding a lot of tests. It is definitely good to have extensive test coverage, but end-to-end tests such as yours add quite a bit to the build time. Flink uses Travis as CI service which kills builds after 2h. Unfortunately, we are experiencing build time outs already and have to be careful when adding tests.
I would like to ask you to remove some of the tests which check for different expression syntax but end up in identical executions. In addition, it would be good to add one SQL test that executes a query which sorts on two fields to have the SQL part covered.
We will add a unit test framework, that checks for correct parsing of the expressions without actually executing queries in a separate effort.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-214862766
I tried to address all your comments. Unfortunately I didn't notice lines longer than 100 yesterday, sorry for that.
I started working on outer joins, but unfortunately I could not find an appropriate jira, so I would be grateful if you could point me to such. Afterwards I will happily work on introducing storages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60923473
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/rules/dataSet/DataSetSortRule.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.rules.dataSet
+
+import org.apache.calcite.plan.{Convention, RelOptRule, RelTraitSet}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.convert.ConverterRule
+import org.apache.calcite.rel.logical.LogicalSort
+import org.apache.flink.api.table.plan.nodes.dataset.{DataSetConvention, DataSetSort}
+
+class DataSetSortRule
+ extends ConverterRule(
+ classOf[LogicalSort],
+ Convention.NONE,
+ DataSetConvention.INSTANCE,
+ "FlinkSortRule") {
+ override def convert(rel: RelNode): RelNode = {
--- End diff --
Please a a new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60929272
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.desc)
+
+ val expected = "3,Third\n2,Second\n1,First"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByAsc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.asc)
+
+ val expected = "1,First\n2,Second\n3,Third"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ @Test
+ def testOrderByMultipleFieldsDifferentDirections(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ env.setParallelism(2)
+ val t = env.fromElements((1, 3, "Third"), (1, 2, "Fourth"), (1, 4, "Second"),
+ (2, 1, "Sixth"), (1, 5, "First"), (1, 1, "Fifth"))
+ .toTable(tEnv).orderBy('_1.asc, '_2.desc)
+
+ val expected = "1,5,First\n1,4,Second\n1,3,Third\n1,2,Fourth\n1,1,Fifth\n2,1,Sixth"
+ val results = t.toDataSet[Row].mapPartition(rows => Seq(rows.toSeq)).collect()
+
+ implicit def rowOrdering = Ordering.by((x : Row) => (x.productElement(0).asInstanceOf[Int],
+ - x.productElement(1).asInstanceOf[Int]))
+
+ val result = results.sortBy(p => p.min).reduceLeft(_ ++ _)
+
+ compareOrderedResultAsText(expected, result)
+ }
+
+ @Test
+ def testOrderByMultipleFieldsWithSql(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val sqlQuery = "SELECT * FROM MyTable ORDER BY _1 DESC, _2 DESC"
+
+ val t = env.fromElements((1, 1, "First"), (2, 3, "Fourth"), (1, 2, "Second"),
+ (2, 1, "Third")).toTable(tEnv)
+ tEnv.registerDataSet("MyTable", t)
+
+ val queryResult = tEnv.sql(sqlQuery)
+
+ val expected = "2,3,Fourth\n2,1,Third\n1,2,Second\n1,1,First"
+ val results = queryResult.toDataSet[Row].setParallelism(1).collect()
+ compareOrderedResultAsText(expected, results)
+ }
+
+ private def compareOrderedResultAsText[T](expected: String, results: Seq[T]) = {
--- End diff --
If we always use `toDataSet[Row]` and respect the expected type in `DataSetSort`, we should not need this method because the result should always be `DataSet[Row]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60924404
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/rules/dataSet/DataSetSortRule.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.rules.dataSet
+
+import org.apache.calcite.plan.{Convention, RelOptRule, RelTraitSet}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.convert.ConverterRule
+import org.apache.calcite.rel.logical.LogicalSort
+import org.apache.flink.api.table.plan.nodes.dataset.{DataSetConvention, DataSetSort}
+
+class DataSetSortRule
--- End diff --
Our implementation of `DataSetSort` does not support offset and limit (which is fine and can be added later, IMO). Consequently, we can only translate `LogicalSort` into `DataSetSort` if `offset` and `limit` are both null.
We need to override the method `matches` and only return `true` if `offset` and `limit` are null. See `DataSetJoinRule` for instance which checks if the join type is `JoinType.INNER`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60927218
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.operators.{DataSource, Operator, PartitionOperator}
+import org.apache.flink.api.table.{BatchTableEnvironment, TableConfig}
+import org.apache.flink.api.table.typeutils.TypeConverter
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+
+ override def copy(traitSet: RelTraitSet, inputs: util.List[RelNode]): RelNode ={
+ new DataSetSort(
+ cluster,
+ traitSet,
+ inputs.get(0),
+ collations,
+ rowType
+ )
+ }
+
+ override def translateToPlan(
+ tableEnv: BatchTableEnvironment,
+ expectedType: Option[TypeInformation[Any]] = None): DataSet[Any] = {
+
+ val config = tableEnv.getConfig
+
+ val inputDS = wrapDataSet(inp.asInstanceOf[DataSetRel].translateToPlan(tableEnv))
+
+ val returnType = determineReturnType(
+ getRowType,
+ expectedType,
+ config.getNullCheck,
+ config.getEfficientTypeUsage
+ )
+
+ val fieldCollations = collations.getFieldCollations.asScala
+ .map(c => (c.getFieldIndex, directionToOrder(c.getDirection)))
+
+ val currentParallelism = if (inputDS.getParallelism >= 1) {
+ inputDS.getParallelism
+ } else {
+ inputDS.getExecutionEnvironment.getParallelism
+ }
+
+ var partitionedDs = if (currentParallelism == 1) {
+ inputDS.setParallelism(1)
+ } else {
+ wrapDataSet(inputDS.partitionByRange(fieldCollations.map(_._1): _*).javaSet
+ .asInstanceOf[PartitionOperator[Any]]
+ .withOrders(fieldCollations.map(_._2): _*))
+ }
+
+ fieldCollations.foreach { fieldCollation =>
+ partitionedDs = partitionedDs.sortPartition(fieldCollation._1, fieldCollation._2)
+ }
+
+ partitionedDs.javaSet
+ }
+
+ private def directionToOrder(direction: Direction) = {
+ direction match {
+ case Direction.ASCENDING | Direction.STRICTLY_ASCENDING => Order.ASCENDING
+ case Direction.DESCENDING | Direction.STRICTLY_DESCENDING => Order.DESCENDING
+ }
+
+ }
+
+ private def wrapDataSet(dataSet: DataSet[Any]) = {
--- End diff --
We do not need to wrap Java `DataSet` into Scala DataSet`. The other `DataSetRel` are also only dealing with Java `DataSet`. This also means we do not need to unwrap it with `javaSet`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61349443
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/rules/dataSet/DataSetSortRule.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.rules.dataSet
+
+import org.apache.calcite.plan.{Convention, RelOptRule, RelOptRuleCall, RelTraitSet}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.convert.ConverterRule
+import org.apache.calcite.rel.core.JoinRelType
+import org.apache.calcite.rel.logical.{LogicalJoin, LogicalSort}
+import org.apache.flink.api.table.plan.nodes.dataset.{DataSetConvention, DataSetSort}
+
+class DataSetSortRule
+ extends ConverterRule(
+ classOf[LogicalSort],
+ Convention.NONE,
+ DataSetConvention.INSTANCE,
+ "FlinkSortRule") {
--- End diff --
"FlinkSortRule" -> "DataSetSortRule"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60926405
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.operators.{DataSource, Operator, PartitionOperator}
+import org.apache.flink.api.table.{BatchTableEnvironment, TableConfig}
+import org.apache.flink.api.table.typeutils.TypeConverter
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+
+ override def copy(traitSet: RelTraitSet, inputs: util.List[RelNode]): RelNode ={
+ new DataSetSort(
+ cluster,
+ traitSet,
+ inputs.get(0),
+ collations,
+ rowType
+ )
+ }
+
+ override def translateToPlan(
+ tableEnv: BatchTableEnvironment,
+ expectedType: Option[TypeInformation[Any]] = None): DataSet[Any] = {
+
+ val config = tableEnv.getConfig
+
+ val inputDS = wrapDataSet(inp.asInstanceOf[DataSetRel].translateToPlan(tableEnv))
+
+ val returnType = determineReturnType(
+ getRowType,
+ expectedType,
+ config.getNullCheck,
+ config.getEfficientTypeUsage
+ )
+
+ val fieldCollations = collations.getFieldCollations.asScala
+ .map(c => (c.getFieldIndex, directionToOrder(c.getDirection)))
+
+ val currentParallelism = if (inputDS.getParallelism >= 1) {
+ inputDS.getParallelism
+ } else {
+ inputDS.getExecutionEnvironment.getParallelism
+ }
+
+ var partitionedDs = if (currentParallelism == 1) {
+ inputDS.setParallelism(1)
--- End diff --
I don't think we should set the parallelism of the input operator to `1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-214388270
Thanks for the PR @dawidwys. I had a few comments. Please let me know if you have any questions.
Regarding a follow task, we have several open issue to extend / improve the Table API. For instance, you could add support for outer joins. That should not be too hard, because the DataSet API supports them natively. Alternatively, we will soon have an interface to define external tables and there will be issues to implement the interface for different storage systems (HBase, ...) or formats (Parquet, ...).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60929017
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.java.tuple.Tuple
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val t = env.fromElements((1, "First"), (3, "Third"), (2, "Second")).toTable(tEnv)
+ .orderBy('_1.desc)
+
+ val expected = "3,Third\n2,Second\n1,First"
+ val results = t.toDataSet[Row].setParallelism(1).collect()
--- End diff --
I think you change the parallelism of the sort operator to `1ยด with this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60926920
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.operators.{DataSource, Operator, PartitionOperator}
+import org.apache.flink.api.table.{BatchTableEnvironment, TableConfig}
+import org.apache.flink.api.table.typeutils.TypeConverter
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+
+ override def copy(traitSet: RelTraitSet, inputs: util.List[RelNode]): RelNode ={
+ new DataSetSort(
+ cluster,
+ traitSet,
+ inputs.get(0),
+ collations,
+ rowType
+ )
+ }
+
+ override def translateToPlan(
+ tableEnv: BatchTableEnvironment,
+ expectedType: Option[TypeInformation[Any]] = None): DataSet[Any] = {
+
+ val config = tableEnv.getConfig
+
+ val inputDS = wrapDataSet(inp.asInstanceOf[DataSetRel].translateToPlan(tableEnv))
+
+ val returnType = determineReturnType(
+ getRowType,
+ expectedType,
+ config.getNullCheck,
+ config.getEfficientTypeUsage
+ )
+
+ val fieldCollations = collations.getFieldCollations.asScala
+ .map(c => (c.getFieldIndex, directionToOrder(c.getDirection)))
+
+ val currentParallelism = if (inputDS.getParallelism >= 1) {
--- End diff --
I am not sure about the handling of the parallelism here. I would only check if the default parallelism of the environment is set to 1. If no, we add a range partition operator, if no, we don't.
Parallelism is not really handled by the Table API at the moment. This is something that we need to improve in the future but not as part of this PR, IMO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60923428
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/table.scala ---
@@ -401,6 +400,52 @@ class Table(
new Table(relBuilder.build(), tableEnv)
}
+ /**
+ * Sorts the given [[Table]]. Similar to SQL ORDER BY. The sorting is performed locally across
+ * all partitions with keys equal to the given fields.
+ *
+ * Example:
+ *
+ * {{{
+ * tab.orderBy('name)
+ * }}}
+ */
+ def orderBy(fields: Expression*): Table = {
+ relBuilder.push(relNode)
+
+ if (! fields.forall {
+ case x: UnresolvedFieldReference => true
+ case x@(_: Asc | _: Desc) => x.asInstanceOf[UnaryExpression]
--- End diff --
With a common base class of `Asc` and `Desc` this can be done nicer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61349569
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/table/test/SortITCase.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.table.test
+
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.scala.util.CollectionDataSets
+import org.apache.flink.api.scala.{ExecutionEnvironment, _}
+import org.apache.flink.api.table.{Row, TableEnvironment}
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase
+import org.apache.flink.api.table.test.utils.TableProgramsTestBase.TableConfigMode
+import org.apache.flink.test.util.MultipleProgramsTestBase.TestExecutionMode
+import org.apache.flink.test.util.TestBaseUtils
+import org.junit._
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
+
+import scala.collection.JavaConverters._
+
+@RunWith(classOf[Parameterized])
+class SortITCase(
+ mode: TestExecutionMode,
+ configMode: TableConfigMode)
+ extends TableProgramsTestBase(mode, configMode) {
+
+ def getExecutionEnvironment = {
+ val env = ExecutionEnvironment.getExecutionEnvironment
+ env.setParallelism(4)
+ env
+ }
+
+ val tupleDataSetStrings = List((1, 1L, "Hi")
+ ,(2, 2L, "Hello")
+ ,(3, 2L, "Hello world")
+ ,(4, 3L, "Hello world, how are you?")
+ ,(5, 3L, "I am fine.")
+ ,(6, 3L, "Luke Skywalker")
+ ,(7, 4L, "Comment#1")
+ ,(8, 4L, "Comment#2")
+ ,(9, 4L, "Comment#3")
+ ,(10, 4L, "Comment#4")
+ ,(11, 5L, "Comment#5")
+ ,(12, 5L, "Comment#6")
+ ,(13, 5L, "Comment#7")
+ ,(14, 5L, "Comment#8")
+ ,(15, 5L, "Comment#9")
+ ,(16, 6L, "Comment#10")
+ ,(17, 6L, "Comment#11")
+ ,(18, 6L, "Comment#12")
+ ,(19, 6L, "Comment#13")
+ ,(20, 6L, "Comment#14")
+ ,(21, 6L, "Comment#15"))
+
+ @Test
+ def testOrderByDesc(): Unit = {
+ val env = getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env, config)
+
+ val ds = CollectionDataSets.get3TupleDataSet(env)
+ val t = ds.toTable(tEnv).orderBy('_1.desc)
+ implicit def rowOrdering[T <: Product] = Ordering.by((x : T) =>
+ - x.productElement(0).asInstanceOf[Int])
+
+ val expected = sortExpectedly(tupleDataSetStrings)
+ val results = t.toDataSet[Row].mapPartition(rows => Seq(rows.toSeq)).collect()
+
+ val result = results.filterNot(_.isEmpty).sortBy(p => p.min).reduceLeft(_ ++ _)
--- End diff --
`p.min` should be `p.head` because p should already be sorted (same applies for the other tests as well).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r61348811
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/nodes/dataset/DataSetSort.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.plan.nodes.dataset
+
+import java.util
+
+import org.apache.calcite.plan.{RelOptCluster, RelTraitSet}
+import org.apache.calcite.rel.RelFieldCollation.Direction
+import org.apache.calcite.rel.`type`.RelDataType
+import org.apache.calcite.rel.{RelCollation, RelNode, RelWriter, SingleRel}
+import org.apache.flink.api.common.operators.Order
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.DataSet
+import org.apache.flink.api.java.typeutils.PojoTypeInfo
+import org.apache.flink.api.table.BatchTableEnvironment
+import org.apache.flink.api.table.typeutils.TypeConverter._
+
+import scala.collection.JavaConverters._
+
+class DataSetSort(
+ cluster: RelOptCluster,
+ traitSet: RelTraitSet,
+ inp: RelNode,
+ collations: RelCollation,
+ rowType2: RelDataType)
+ extends SingleRel(cluster, traitSet, inp)
+ with DataSetRel{
+
+ override def copy(traitSet: RelTraitSet, inputs: util.List[RelNode]): RelNode ={
+ new DataSetSort(
+ cluster,
+ traitSet,
+ inputs.get(0),
+ collations,
+ rowType2
+ )
+ }
+
+ override def translateToPlan(
+ tableEnv: BatchTableEnvironment,
+ expectedType: Option[TypeInformation[Any]] = None): DataSet[Any] = {
+
+ val config = tableEnv.getConfig
+
+ val inputDS = inp.asInstanceOf[DataSetRel].translateToPlan(tableEnv)
+
+ val currentParallelism = inputDS.getExecutionEnvironment.getParallelism
+ var partitionedDs = if (currentParallelism == 1) {
+ inputDS
+ } else {
+ inputDS.partitionByRange(fieldCollations.map(_._1): _*)
+ .withOrders(fieldCollations.map(_._2): _*)
+ }
+
+ fieldCollations.foreach { fieldCollation =>
+ partitionedDs = partitionedDs.sortPartition(fieldCollation._1, fieldCollation._2)
+ }
+
+ val inputType = partitionedDs.getType
+ expectedType match {
+
+ case None if config.getEfficientTypeUsage =>
+ partitionedDs
+
+ case _ =>
+ val determinedType = determineReturnType(
+ getRowType,
+ expectedType,
+ config.getNullCheck,
+ config.getEfficientTypeUsage)
+
+ // conversion
+ if (determinedType != inputType) {
+
+ val mapFunc = getConversionMapper(config,
+ partitionedDs.getType,
+ determinedType,
+ "DataSetSortConversion",
+ getRowType.getFieldNames.asScala
+ )
+
+ partitionedDs.map(mapFunc)
+ }
+ // no conversion necessary, forward
+ else {
+ partitionedDs
+ }
+ }
+ }
+
+ private def directionToOrder(direction: Direction) = {
+ direction match {
+ case Direction.ASCENDING | Direction.STRICTLY_ASCENDING => Order.ASCENDING
+ case Direction.DESCENDING | Direction.STRICTLY_DESCENDING => Order.DESCENDING
+ }
+
+ }
+
+ private val fieldCollations = collations.getFieldCollations.asScala
+ .map(c => (c.getFieldIndex, directionToOrder(c.getDirection)))
+
+ private val sortFieldsToString = fieldCollations
+ .map(col => s"${rowType2.getFieldNames.get(col._1)} ${col._2.getShortName}" ).mkString(", ")
+
+ override def toString: String = s"Sort(by: $sortFieldsToString)"
+
+ override def explainTerms(pw: RelWriter) : RelWriter = {
+ super.explainTerms(pw)
+ .item("by", sortFieldsToString)
--- End diff --
"by" -> "orderBy"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1926#discussion_r60923266
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/expressions/ordering.scala ---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.table.expressions
+import org.apache.calcite.rex.RexNode
+import org.apache.calcite.tools.RelBuilder
+
+case class Asc(child: Expression) extends UnaryExpression {
--- End diff --
You add a common base class for orderings which `Asc` and `Desc` extend?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-215579092
Merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1926#issuecomment-215254090
Thanks for the update @dawidwys. PR looks good except for a few minor comments.
Can you also rebase to the current master? A recent commit updated some code that your PR is touching (`DataSetScan`).
Then it should be good to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2946] Add orderBy() to Table API
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1926
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---