You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by fhueske <gi...@git.apache.org> on 2016/04/14 15:56:45 UTC
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/1887
[FLINK-3738] [tableAPI] Refactor TableEnvironments. Remove Translators and TranslationContext.
- [X] General
- The pull request references the related JIRA issue
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message
- [X] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [X] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
- Removes Translators (JavaBatchTranslator, ScalaBatchTranslator, JavaStreamTranslator, ScalaStreamTranslator)
- Removes TranslationContext
- Creates TableEnvironments for Java/Scala DataSet/DataStream APIs
- Functionality of translators and translation context is moved to table environments
- Tables are bound to a TableEnvironment
- Simplifies table translation from and to DataSet / DataStream.
- Updates documentation and cleaned up ScalaDocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink tableEnv
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1887.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1887
----
commit f9f7f1dfb00036d903e0e0d514be52000c26a826
Author: Fabian Hueske <fh...@apache.org>
Date: 2016-04-12T16:48:38Z
[FLINK-3738] [tableAPI] Refactor TableEnvironments. Remove Translators and TranslationContext.
- Removes Translators (JavaBatchTranslator, ScalaBatchTranslator, JavaStreamTranslator, ScalaStreamTranslator)
- Removes TranslationContext
- Creates TableEnvironments for Java/Scala DataSet/DataStream APIs
- Functionality of translators and translation context is moved to table environments
- Tables are bound to a TableEnvironment
- Simplifies table translation from and to DataSet / DataStream.
- Updates documentation and cleaned up ScalaDocs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1887
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the pull request:
https://github.com/apache/flink/pull/1887#issuecomment-209972903
Thanks for the massive refactoring @fhueske!
I looked over the changes and couldn't spot anything noteworthy. The only things that I don't like is the additional `TableEnvironment` creattion even for simple table programs when using implicit conversions. Couldn't we just also provide a static TableEnvironment if we call the parameterless `toTable` that just calls `(Stream)ExecutionEnvironment.getExecutionEnvironment()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/1887#issuecomment-210003346
Just a couple of typos from my side.
I really like it that we got rid of the `TranslationContext` and I'm in favor of the explicit relationship between Tables and TableEnvironment. +1 to merge!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on a diff in the pull request:
https://github.com/apache/flink/pull/1887#discussion_r59735429
--- Diff: docs/apis/batch/libs/table.md ---
@@ -391,9 +375,7 @@ val result = in.distinct();
{% top %}
### Expression Syntax
-Some of operators in previous section expect an expression. These can either be specified using an embedded Scala DSL or
-a String expression. Please refer to the examples above to learn how expressions can be
-formulated.
+Some of operators in previous section expect one or more expressions. Expressions can be specified using an embedded Scala DSL or as Strings. Please refer to the examples above to learn how expressions can be specified.
--- End diff --
Some of _the_ operators or Some operators
*sections
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on a diff in the pull request:
https://github.com/apache/flink/pull/1887#discussion_r59735189
--- Diff: docs/apis/batch/libs/table.md ---
@@ -61,58 +59,73 @@ Note that the Table API is currently not part of the binary distribution. See li
Table API
----------
-The Table API provides methods for running relational operations on Tables, both in Scala and Java.
-In the following sections you can find examples that show how to create Tables, how to define and execute relational queries on them,
-and how to retrieve the result of a query as a `DataSet`.
+The Table API provides methods to apply relational operations on DataSets, both in Scala and Java.
+
+The central concept of the Table API is a `Table` which is represents a table with relational schema (or relation). Tables can be created from a `DataSet`, converted into a `DataSet`, or registered in a table catalog using a `TableEnvironment`. A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine Tables of different TableEnvironments.
--- End diff --
Remove "is".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by vasia <gi...@git.apache.org>.
Github user vasia commented on a diff in the pull request:
https://github.com/apache/flink/pull/1887#discussion_r59735309
--- Diff: docs/apis/batch/libs/table.md ---
@@ -61,58 +59,73 @@ Note that the Table API is currently not part of the binary distribution. See li
Table API
----------
-The Table API provides methods for running relational operations on Tables, both in Scala and Java.
-In the following sections you can find examples that show how to create Tables, how to define and execute relational queries on them,
-and how to retrieve the result of a query as a `DataSet`.
+The Table API provides methods to apply relational operations on DataSets, both in Scala and Java.
+
+The central concept of the Table API is a `Table` which is represents a table with relational schema (or relation). Tables can be created from a `DataSet`, converted into a `DataSet`, or registered in a table catalog using a `TableEnvironment`. A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine Tables of different TableEnvironments.
+
+The following sections show by example how to use the Table API embedded in the Scala and Java DataSet APIs.
### Scala Table API
-The Table API can be enabled by importing `org.apache.flink.api.scala.table._`. This enables
-implicit conversions that allow
-converting a DataSet to a Table. This example shows how a DataSet can
-be converted, how relational queries can be specified and how a Table can be
-converted back to a DataSet:
+The Table API is enabled by importing `org.apache.flink.api.scala.table._`. This enables
+implicit conversions to convert a DataSet to a Table. The following example shows:
+
+- how a `DataSet` is converted to a `Table`,
+- how relational queries are specified, and
+- how a `Table` is converted back to a `DataSet`.
{% highlight scala %}
import org.apache.flink.api.scala._
import org.apache.flink.api.scala.table._
case class WC(word: String, count: Int)
+
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tEnv = TableEnvironment.getTableEnvironment(env)
+
val input = env.fromElements(WC("hello", 1), WC("hello", 1), WC("ciao", 1))
-val expr = input.toTable
-val result = expr.groupBy('word).select('word, 'count.sum as 'count).toDataSet[WC]
+val expr = input.toTable(tEnv)
+val result = expr
+ .groupBy('word)
+ .select('word, 'count.sum as 'count)
+ .toDataSet[WC]
{% endhighlight %}
The expression DSL uses Scala symbols to refer to field names and code generation to
transform expressions to efficient runtime code. Please note that the conversion to and from
-Tables only works when using Scala case classes or Flink POJOs. Please check out
-the [Type Extraction and Serialization]({{ site.baseurl }}/internals/types_serialization.html) section
-to learn the requirements for a class to be considered a POJO.
+Tables only works when using Scala case classes or Java POJOs. Please refer to the [Type Extraction and Serialization]({{ site.baseurl }}/internals/types_serialization.html) section
+to learn the characteristics of a valid POJO.
-This is another example that shows how you
-can join two Tables:
+Another example shows how to join two Tables:
{% highlight scala %}
case class MyResult(a: String, d: Int)
-val input1 = env.fromElements(...).toTable.as('a, 'b)
-val input2 = env.fromElements(...).toTable.as('c, 'd)
-val joined = input1.join(input2).where("a = c && d > 42").select("a, d").toDataSet[MyResult]
+val input1 = env.fromElements(...).toTable(tEnv).as('a, 'b)
+val input2 = env.fromElements(...).toTable(tEnv, 'c, 'd)
+
+val joined = input1.join(input2)
+ .where("a = c && d > 42")
+ .select("a, d")
+ .toDataSet[MyResult]
{% endhighlight %}
-Notice, how a DataSet can be converted to a Table by using `as` and specifying new
-names for the fields. This can also be used to disambiguate fields before a join operation. Also,
-in this example we see that you can also use Strings to specify relational expressions.
+Notice, how the field names of a Table can be changed with `as()` or specified with `toTable()` when converting a DataSet to a Table. In addition, the example shows how to use Strings to specify relational expressions.
-Please refer to the Scaladoc (and Javadoc) for a full list of supported operations and a
-description of the expression syntax.
+Please refer to the Scaladoc (and Javadoc) for a full list of supported operations and a description of the expression syntax.
{% top %}
### Java Table API
-When using Java, Tables can be converted to and from DataSet using `TableEnvironment`.
-This example is equivalent to the above Scala Example:
+When using Flink's Java DataSet API, DataSet are converted to Tables and Tables to DataSet using a `TableEnvironment`.
--- End diff --
*DataSets
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1887#issuecomment-209979690
I thought about that as well, but decided against an implicit conversion without TableEnvironment.
What might happen otherwise is that users try to join or union tables that have different implicitly created TableEnvironments or to register an implicitly converted table in a TableEnvironment. This does not work due to Calcite internals. I think it is better to make the Table/TableEnvironment relationship explicit and also easier to understand for users if an exception is thrown for the above reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1887#issuecomment-210012116
Thanks for the review @vasia.
I fixed the typos.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3738] [tableAPI] Refactor TableEnvironm...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1887#issuecomment-210399072
will merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---