You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by sj...@apache.org on 2020/12/09 21:42:16 UTC
[flink] 01/02: Revert "[FLINK-20551][docs] Remove legacy planner
from index and common concepts"
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch release-1.12
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 6132b8c67e3a5253fb92f43b72b5a4da5482384c
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Wed Dec 9 15:32:40 2020 -0600
Revert "[FLINK-20551][docs] Remove legacy planner from index and common concepts"
This reverts commit f29836bc7abd28b44cc79e1a150e4bcc49887d0b.
---
docs/dev/python/table_api_tutorial.md | 2 +-
docs/dev/table/common.md | 482 ++++++++++++++++++++++------------
docs/dev/table/index.md | 22 +-
docs/dev/table/legacy_planner.md | 274 -------------------
docs/redirects/temporal_table.md | 2 +-
5 files changed, 341 insertions(+), 441 deletions(-)
diff --git a/docs/dev/python/table_api_tutorial.md b/docs/dev/python/table_api_tutorial.md
index 5003e69..9a7ab5e 100644
--- a/docs/dev/python/table_api_tutorial.md
+++ b/docs/dev/python/table_api_tutorial.md
@@ -71,7 +71,7 @@ t_config = TableConfig()
t_env = BatchTableEnvironment.create(exec_env, t_config)
{% endhighlight %}
-The table environment created, you can declare source and sink tables.
+The the table environment created, you can declare source and sink tables.
{% highlight python %}
t_env.connect(FileSystem().path('/tmp/input')) \
diff --git a/docs/dev/table/common.md b/docs/dev/table/common.md
index 6bcc295..2f8c452 100644
--- a/docs/dev/table/common.md
+++ b/docs/dev/table/common.md
@@ -22,23 +22,33 @@ specific language governing permissions and limitations
under the License.
-->
-The Table API and SQL are integrated in a joint API.
-The central concept of this API is a `Table` which serves as input and output of queries.
-This document shows the common structure of programs with Table API and SQL queries, how to register a `Table`, how to query a `Table`, and how to emit a `Table`.
+The Table API and SQL are integrated in a joint API. The central concept of this API is a `Table` which serves as input and output of queries. This document shows the common structure of programs with Table API and SQL queries, how to register a `Table`, how to query a `Table`, and how to emit a `Table`.
* This will be replaced by the TOC
{:toc}
+Main Differences Between the Two Planners
+-----------------------------------------
+
+1. Blink treats batch jobs as a special case of streaming. As such, the conversion between Table and DataSet is also not supported, and batch jobs will not be translated into `DateSet` programs but translated into `DataStream` programs, the same as the streaming jobs.
+2. The Blink planner does not support `BatchTableSource`, use bounded `StreamTableSource` instead of it.
+3. The implementations of `FilterableTableSource` for the old planner and the Blink planner are incompatible. The old planner will push down `PlannerExpression`s into `FilterableTableSource`, while the Blink planner will push down `Expression`s.
+4. String based key-value config options (Please see the documentation about [Configuration]({% link dev/table/config.md %}) for details) are only used for the Blink planner.
+5. The implementation(`CalciteConfig`) of `PlannerConfig` in two planners is different.
+6. The Blink planner will optimize multiple-sinks into one DAG on both `TableEnvironment` and `StreamTableEnvironment`. The old planner will always optimize each sink into a new DAG, where all DAGs are independent of each other.
+7. The old planner does not support catalog statistics now, while the Blink planner does.
+
+
Structure of Table API and SQL Programs
---------------------------------------
-The following code example shows the common structure of Table API and SQL programs.
+All Table API and SQL programs for batch and streaming follow the same pattern. The following code example shows the common structure of Table API and SQL programs.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-// create a TableEnvironment for batch or streaming execution
+// create a TableEnvironment for specific planner batch or streaming
TableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
// create a Table
@@ -61,7 +71,7 @@ tableResult...
<div data-lang="scala" markdown="1">
{% highlight scala %}
-// create a TableEnvironment for batch or streaming execution
+// create a TableEnvironment for specific planner batch or streaming
val tableEnv = ... // see "Create a TableEnvironment" section
// create a Table
@@ -84,7 +94,7 @@ tableResult...
<div data-lang="python" markdown="1">
{% highlight python %}
-# create a TableEnvironment for batch or streaming execution
+# create a TableEnvironment for specific planner batch or streaming
table_env = ... # see "Create a TableEnvironment" section
# register a Table
@@ -106,119 +116,173 @@ table_result...
</div>
</div>
-**Note:** Table API and SQL queries can be easily integrated with and embedded into DataStream programs.
-Have a look at the [Integration with DataStream](#integration-with-datastream) section to learn how DataStreams can be converted into Tables and vice versa.
+**Note:** Table API and SQL queries can be easily integrated with and embedded into DataStream or DataSet programs. Have a look at the [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api) section to learn how DataStreams and DataSets can be converted into Tables and vice versa.
{% top %}
Create a TableEnvironment
-------------------------
-The `TableEnvironment` is the entrypoint for Table API and SQL integration and is responsible for:
+The `TableEnvironment` is a central concept of the Table API and SQL integration. It is responsible for:
* Registering a `Table` in the internal catalog
* Registering catalogs
* Loading pluggable modules
* Executing SQL queries
* Registering a user-defined (scalar, table, or aggregation) function
-* Converting a `DataStream` into a `Table`
-* Holding a reference to a `StreamExecutionEnvironment`
+* Converting a `DataStream` or `DataSet` into a `Table`
+* Holding a reference to an `ExecutionEnvironment` or `StreamExecutionEnvironment`
+
+A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine tables of different TableEnvironments in the same query, e.g., to join or union them.
+
+A `TableEnvironment` is created by calling the static `BatchTableEnvironment.create()` or `StreamTableEnvironment.create()` method with a `StreamExecutionEnvironment` or an `ExecutionEnvironment` and an optional `TableConfig`. The `TableConfig` can be used to configure the `TableEnvironment` or to customize the query optimization and translation process (see [Query Optimization](#query-optimization)).
-A `Table` is always bound to a specific `TableEnvironment`.
-It is not possible to combine tables of different TableEnvironments in the same query, e.g., to join or union them.
-A `TableEnvironment` is created by calling the static `TableEnvironment.create()` method.
+Make sure to choose the specific planner `BatchTableEnvironment`/`StreamTableEnvironment` that matches your programming language.
+
+If both planner jars are on the classpath (the default behavior), you should explicitly set which planner to use in the current program.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-import org.apache.flink.table.api.EnvironmentSettings;
-import org.apache.flink.table.api.TableEnvironment;
-EnvironmentSettings settings = EnvironmentSettings
- .newInstance()
- .inStreamingMode()
- //.inBatchMode()
- .build();
+// **********************
+// FLINK STREAMING QUERY
+// **********************
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
-TableEnvironment tEnv = TableEnvironment.create(setting);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
+EnvironmentSettings fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build();
+StreamExecutionEnvironment fsEnv = StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSettings);
+// or TableEnvironment fsTableEnv = TableEnvironment.create(fsSettings);
-val settings = EnvironmentSettings
- .newInstance()
- .inStreamingMode()
- //.inBatchMode()
- .build()
+// ******************
+// FLINK BATCH QUERY
+// ******************
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.table.api.bridge.java.BatchTableEnvironment;
-val tEnv = TableEnvironment.create(setting)
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-# ***************
-# Streaming QUERY
-# ***************
+ExecutionEnvironment fbEnv = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment fbTableEnv = BatchTableEnvironment.create(fbEnv);
-from pyflink.datastream import StreamExecutionEnvironment
-from pyflink.table import StreamTableEnvironment, EnvironmentSettings
+// **********************
+// BLINK STREAMING QUERY
+// **********************
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
-b_s_env = StreamExecutionEnvironment.get_execution_environment()
-b_s_settings = EnvironmentSettings.new_instance().use_blink_planner().in_streaming_mode().build()
-b_s_t_env = StreamTableEnvironment.create(b_s_env, environment_settings=b_s_settings)
+StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment();
+EnvironmentSettings bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
+StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);
+// or TableEnvironment bsTableEnv = TableEnvironment.create(bsSettings);
-# ***********
-# BATCH QUERY
-# ***********
+// ******************
+// BLINK BATCH QUERY
+// ******************
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
-from pyflink.table import EnvironmentSettings, BatchTableEnvironment
+EnvironmentSettings bbSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
+TableEnvironment bbTableEnv = TableEnvironment.create(bbSettings);
-b_b_settings = EnvironmentSettings.new_instance().use_blink_planner().in_batch_mode().build()
-b_b_t_env = BatchTableEnvironment.create(environment_settings=b_b_settings)
{% endhighlight %}
</div>
-</div>
-Alternatively, users can create a `StreamTableEnvironment` from an existing `StreamExecutionEnvironment`
-to interoperate with the `DataStream` API.
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.table.api.EnvironmentSettings;
-import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+// **********************
+// FLINK STREAMING QUERY
+// **********************
+import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
+import org.apache.flink.table.api.EnvironmentSettings
+import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+val fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build()
+val fsEnv = StreamExecutionEnvironment.getExecutionEnvironment
+val fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSettings)
+// or val fsTableEnv = TableEnvironment.create(fsSettings)
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
+// ******************
+// FLINK BATCH QUERY
+// ******************
+import org.apache.flink.api.scala.ExecutionEnvironment
+import org.apache.flink.table.api.bridge.scala.BatchTableEnvironment
+
+val fbEnv = ExecutionEnvironment.getExecutionEnvironment
+val fbTableEnv = BatchTableEnvironment.create(fbEnv)
+
+// **********************
+// BLINK STREAMING QUERY
+// **********************
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.table.api.EnvironmentSettings
import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tEnv = StreamTableEnvironment.create(env)
+val bsEnv = StreamExecutionEnvironment.getExecutionEnvironment
+val bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build()
+val bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings)
+// or val bsTableEnv = TableEnvironment.create(bsSettings)
+
+// ******************
+// BLINK BATCH QUERY
+// ******************
+import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
+
+val bbSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build()
+val bbTableEnv = TableEnvironment.create(bbSettings)
+
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
+
+# **********************
+# FLINK STREAMING QUERY
+# **********************
+from pyflink.datastream import StreamExecutionEnvironment
+from pyflink.table import StreamTableEnvironment, EnvironmentSettings
+
+f_s_env = StreamExecutionEnvironment.get_execution_environment()
+f_s_settings = EnvironmentSettings.new_instance().use_old_planner().in_streaming_mode().build()
+f_s_t_env = StreamTableEnvironment.create(f_s_env, environment_settings=f_s_settings)
+
+# ******************
+# FLINK BATCH QUERY
+# ******************
+from pyflink.dataset import ExecutionEnvironment
+from pyflink.table import BatchTableEnvironment
+
+f_b_env = ExecutionEnvironment.get_execution_environment()
+f_b_t_env = BatchTableEnvironment.create(f_b_env, table_config)
+
+# **********************
+# BLINK STREAMING QUERY
+# **********************
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment, EnvironmentSettings
b_s_env = StreamExecutionEnvironment.get_execution_environment()
b_s_settings = EnvironmentSettings.new_instance().use_blink_planner().in_streaming_mode().build()
b_s_t_env = StreamTableEnvironment.create(b_s_env, environment_settings=b_s_settings)
+
+# ******************
+# BLINK BATCH QUERY
+# ******************
+from pyflink.table import EnvironmentSettings, BatchTableEnvironment
+
+b_b_settings = EnvironmentSettings.new_instance().use_blink_planner().in_batch_mode().build()
+b_b_t_env = BatchTableEnvironment.create(environment_settings=b_b_settings)
+
{% endhighlight %}
</div>
</div>
+**Note:** If there is only one planner jar in `/lib` directory, you can use `useAnyPlanner` (`use_any_planner` for python) to create specific `EnvironmentSettings`.
+
{% top %}
Create Tables in the Catalog
@@ -753,8 +817,11 @@ Translate and Execute a Query
-----------------------------
The behavior of translating and executing a query is different for the two planners.
-Table API and SQL queries are translated into [DataStream]({% link dev/datastream_api.md %}) programs whether their input is streaming or batch.
-A query is internally represented as a logical query plan and is translated in two phases:
+
+<div class="codetabs" markdown="1">
+
+<div data-lang="Blink planner" markdown="1">
+Table API and SQL queries are translated into [DataStream]({% link dev/datastream_api.md %}) programs whether their input is streaming or batch. A query is internally represented as a logical query plan and is translated in two phases:
1. Optimization of the logical plan,
2. Translation into a DataStream program.
@@ -765,35 +832,59 @@ a Table API or SQL query is translated when:
* `Table.executeInsert()` is called. This method is used for inserting the table content to the given sink path, and the Table API is translated immediately once this method is called.
* `Table.execute()` is called. This method is used for collecting the table content to local client, and the Table API is translated immediately once this method is called.
* `StatementSet.execute()` is called. A `Table` (emitted to a sink through `StatementSet.addInsert()`) or an INSERT statement (specified through `StatementSet.addInsertSql()`) will be buffered in `StatementSet` first. They are translated once `StatementSet.execute()` is called. All sinks will be optimized into one DAG.
-* A `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream](#integration-with-datastream)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called.
+* A `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called.
<span class="label label-danger">Attention</span> **Since 1.11 version, `sqlUpdate()` method and `insertInto()` method are deprecated. If the Table program is built from these two methods, we must use `StreamTableEnvironment.execute()` method instead of `StreamExecutionEnvironment.execute()` method to execute it.**
+</div>
+
+<div data-lang="Old planner" markdown="1">
+Table API and SQL queries are translated into [DataStream]({% link dev/datastream_api.md %}) or [DataSet]({% link dev/batch/index.md %}) programs depending on whether their input is a streaming or batch input. A query is internally represented as a logical query plan and is translated in two phases:
+
+1. Optimization of the logical plan
+2. Translation into a DataStream or DataSet program
+
+A Table API or SQL query is translated when:
+
+* `TableEnvironment.executeSql()` is called. This method is used for executing a given statement, and the sql query is translated immediately once this method is called.
+* `Table.executeInsert()` is called. This method is used for inserting the table content to the given sink path, and the Table API is translated immediately once this method is called.
+* `Table.execute()` is called. This method is used for collecting the table content to local client, and the Table API is translated immediately once this method is called.
+* `StatementSet.execute()` is called. A `Table` (emitted to a sink through `StatementSet.addInsert()`) or an INSERT statement (specified through `StatementSet.addInsertSql()`) will be buffered in `StatementSet` first. They are translated once `StatementSet.execute()` is called. Each sink will be optimized independently. The execution graph contains multiple independent sub-DAGs.
+* For streaming, a `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called. For batch, a `Table` is translated when it is converted into a `DataSet` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, [...]
+
+<span class="label label-danger">Attention</span> **Since 1.11 version, `sqlUpdate()` method and `insertInto()` method are deprecated. For streaming, if the Table program is built from these two methods, we must use `StreamTableEnvironment.execute()` method instead of `StreamExecutionEnvironment.execute()` method to execute it. For batch, if the Table program is built from these two methods, we must use `BatchTableEnvironment.execute()` method instead of `ExecutionEnvironment.execute()` [...]
+
+</div>
+
+</div>
+
{% top %}
-Integration with DataStream
----------------------------
+Integration with DataStream and DataSet API
+-------------------------------------------
+
+Both planners on stream can integrate with the `DataStream` API. Only old planner can integrate with the `DataSet API`, Blink planner on batch could not be combined with both.
+**Note:** The `DataSet` API discussed below is only relevant for the old planner on batch.
-Table API and SQL queries can be easily integrated with and embedded into [DataStream]({% link dev/datastream_api.md %}) programs.
-For instance, it is possible to query an external table (for example from a RDBMS), do some pre-processing, such as filtering, projecting, aggregating, or joining with metadata,
-and then further process the data with the DataStream API.
-Inversely, a Table API or SQL query can also be applied on the result of a DataStream program.
+Table API and SQL queries can be easily integrated with and embedded into [DataStream]({% link dev/datastream_api.md %}) and [DataSet]({% link dev/batch/index.md %}) programs. For instance, it is possible to query an external table (for example from a RDBMS), do some pre-processing, such as filtering, projecting, aggregating, or joining with meta data, and then further process the data with either the DataStream or DataSet API (and any of the libraries built on top of these APIs, such as [...]
+
+This interaction can be achieved by converting a `DataStream` or `DataSet` into a `Table` and vice versa. In this section, we describe how these conversions are done.
### Implicit Conversion for Scala
-The Scala Table API features implicit conversions for `DataStream` and `Table` classes.
-These conversions are enabled by importing the package `org.apache.flink.table.api.bridge.scala._` in addition to `org.apache.flink.api.scala._` for the Scala DataStream API.
+The Scala Table API features implicit conversions for the `DataSet`, `DataStream`, and `Table` classes. These conversions are enabled by importing the package `org.apache.flink.table.api.bridge.scala._` in addition to `org.apache.flink.api.scala._` for the Scala DataStream API.
-### Create a View from a DataStream
+### Create a View from a DataStream or DataSet
-A `DataStream` can be registered in a `TableEnvironment` as a View and the schema is dependent only the `DataStream`s underlying data type.
-Please see the section about [mapping of data types to table schema](#mapping-of-data-types-to-table-schema) for details.
+A `DataStream` or `DataSet` can be registered in a `TableEnvironment` as a View. The schema of the resulting view depends on the data type of the registered `DataStream` or `DataSet`. Please check the section about [mapping of data types to table schema](#mapping-of-data-types-to-table-schema) for details.
-**Note:** Views created from a `DataStream` can only be registered as temporary views.
+**Note:** Views created from a `DataStream` or `DataSet` can be registered as temporary views only.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
+// get StreamTableEnvironment
+// registration of a DataSet in a BatchTableEnvironment is equivalent
StreamTableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
DataStream<Tuple2<Long, String>> stream = ...
@@ -805,42 +896,59 @@ tableEnv.createTemporaryView("myTable", stream);
tableEnv.createTemporaryView("myTable2", stream, $("myLong"), $("myString"));
{% endhighlight %}
</div>
+
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val tableEnv: StreamTableEnvironment = ???
+// get TableEnvironment
+// registration of a DataSet is equivalent
+val tableEnv: StreamTableEnvironment = ... // see "Create a TableEnvironment" section
val stream: DataStream[(Long, String)] = ...
-// Register the DataStream as View "myTable" with fields "f0", "f1"
+// register the DataStream as View "myTable" with fields "f0", "f1"
tableEnv.createTemporaryView("myTable", stream)
-// Register the DataStream as View "myTable2" with fields "myLong", "myString"
-tableEnv.createTemporaryView("myTable2", stream, $"myLong", $"myString");
+// register the DataStream as View "myTable2" with fields "myLong", "myString"
+tableEnv.createTemporaryView("myTable2", stream, 'myLong, 'myString)
{% endhighlight %}
</div>
</div>
{% top %}
-### Convert a DataStream into a Table
+### Convert a DataStream or DataSet into a Table
-A `DataStream` can be directly converted to a `Table` in a `StreamTableEnvironment`.
-The schema of the resulting view depends on the data type of the registered collection.
+Instead of registering a `DataStream` or `DataSet` in a `TableEnvironment`, it can also be directly converted into a `Table`. This is convenient if you want to use the Table in a Table API query.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-StreamTableEnvironment tableEnv = ...;
+// get StreamTableEnvironment
+// registration of a DataSet in a BatchTableEnvironment is equivalent
+StreamTableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
+
DataStream<Tuple2<Long, String>> stream = ...
+// Convert the DataStream into a Table with default fields "f0", "f1"
+Table table1 = tableEnv.fromDataStream(stream);
+
+// Convert the DataStream into a Table with fields "myLong", "myString"
Table table2 = tableEnv.fromDataStream(stream, $("myLong"), $("myString"));
{% endhighlight %}
</div>
+
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val tableEnv: StreamTableEnvironment = ???
-val stream: DataStream[(Long, String)] = ???
+// get TableEnvironment
+// registration of a DataSet is equivalent
+val tableEnv = ... // see "Create a TableEnvironment" section
+
+val stream: DataStream[(Long, String)] = ...
+// convert the DataStream into a Table with default fields "_1", "_2"
+val table1: Table = tableEnv.fromDataStream(stream)
+
+// convert the DataStream into a Table with fields "myLong", "myString"
val table2: Table = tableEnv.fromDataStream(stream, $"myLong", $"myString")
{% endhighlight %}
</div>
@@ -848,14 +956,11 @@ val table2: Table = tableEnv.fromDataStream(stream, $"myLong", $"myString")
{% top %}
-### Convert a Table into a DataStream
+### Convert a Table into a DataStream or DataSet
-A results of a `Table` can be converted into a `DataStream`.
-In this way, custom DataStream programs can be run on the result of a Table API or SQL query.
+A `Table` can be converted into a `DataStream` or `DataSet`. In this way, custom DataStream or DataSet programs can be run on the result of a Table API or SQL query.
-When converting a `Table` into a `DataStream` you need to specify the data type of the resulting records, i.e., the data type into which the rows of the `Table` are to be converted.
-Often the most convenient conversion type is `Row`.
-The following list gives an overview of the features of the different options:
+When converting a `Table` into a `DataStream` or `DataSet`, you need to specify the data type of the resulting `DataStream` or `DataSet`, i.e., the data type into which the rows of the `Table` are to be converted. Often the most convenient conversion type is `Row`. The following list gives an overview of the features of the different options:
- **Row**: fields are mapped by position, arbitrary number of fields, support for `null` values, no type-safe access.
- **POJO**: fields are mapped by name (POJO fields must be named as `Table` fields), arbitrary number of fields, support for `null` values, type-safe access.
@@ -875,54 +980,53 @@ There are two modes to convert a `Table` into a `DataStream`:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-StreamTableEnvironment tableEnv = ...;
+// get StreamTableEnvironment.
+StreamTableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
-Table table = tableEnv.fromValues(
- DataTypes.Row(
- DataTypes.FIELD("name", DataTypes.STRING()),
- DataTypes.FIELD("age", DataTypes.INT()),
- row("john", 35),
- row("sarah", 32));
+// Table with two fields (String name, Integer age)
+Table table = ...
-// Convert the Table into an append DataStream of Row by specifying the class
+// convert the Table into an append DataStream of Row by specifying the class
DataStream<Row> dsRow = tableEnv.toAppendStream(table, Row.class);
-// Convert the Table into an append DataStream of Tuple2<String, Integer> with TypeInformation
-TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(Types.STRING(), Types.INT());
-DataStream<Tuple2<String, Integer>> dsTuple = tableEnv.toAppendStream(table, tupleType);
-
-// Convert the Table into a retract DataStream of Row.
-// A retract stream of type X is a DataStream<Tuple2<Boolean, X>>.
-// The boolean field indicates the type of the change.
-// True is INSERT, false is DELETE.
-DataStream<Tuple2<Boolean, Row>> retractStream = tableEnv.toRetractStream(table, Row.class);
+// convert the Table into an append DataStream of Tuple2<String, Integer>
+// via a TypeInformation
+TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(
+ Types.STRING(),
+ Types.INT());
+DataStream<Tuple2<String, Integer>> dsTuple =
+ tableEnv.toAppendStream(table, tupleType);
+
+// convert the Table into a retract DataStream of Row.
+// A retract stream of type X is a DataStream<Tuple2<Boolean, X>>.
+// The boolean field indicates the type of the change.
+// True is INSERT, false is DELETE.
+DataStream<Tuple2<Boolean, Row>> retractStream =
+ tableEnv.toRetractStream(table, Row.class);
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val tableEnv: StreamTableEnvironment = ???
+// get TableEnvironment.
+// registration of a DataSet is equivalent
+val tableEnv: StreamTableEnvironment = ... // see "Create a TableEnvironment" section
// Table with two fields (String name, Integer age)
-val table: Table = tableEnv.fromValues(
- DataTypes.Row(
- DataTypes.FIELD("name", DataTypes.STRING()),
- DataTypes.FIELD("age", DataTypes.INT()),
- row("john", 35),
- row("sarah", 32));
-
-// Convert the Table into an append DataStream of Row by specifying the class
+val table: Table = ...
+
+// convert the Table into an append DataStream of Row
val dsRow: DataStream[Row] = tableEnv.toAppendStream[Row](table)
-// Convert the Table into an append DataStream of (String, Integer) with TypeInformation
+// convert the Table into an append DataStream of Tuple2[String, Int]
val dsTuple: DataStream[(String, Int)] dsTuple =
tableEnv.toAppendStream[(String, Int)](table)
-// Convert the Table into a retract DataStream of Row.
-// A retract stream of type X is a DataStream<Tuple2<Boolean, X>>.
-// The boolean field indicates the type of the change.
-// True is INSERT, false is DELETE.
+// convert the Table into a retract DataStream of Row.
+// A retract stream of type X is a DataStream[(Boolean, X)].
+// The boolean field indicates the type of the change.
+// True is INSERT, false is DELETE.
val retractStream: DataStream[(Boolean, Row)] = tableEnv.toRetractStream[Row](table)
{% endhighlight %}
</div>
@@ -932,12 +1036,56 @@ val retractStream: DataStream[(Boolean, Row)] = tableEnv.toRetractStream[Row](ta
<span class="label label-danger">Attention</span> **Once the Table is converted to a DataStream, please use the `StreamExecutionEnvironment.execute()` method to execute the DataStream program.**
+#### Convert a Table into a DataSet
+
+A `Table` is converted into a `DataSet` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get BatchTableEnvironment
+BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env);
+
+// Table with two fields (String name, Integer age)
+Table table = ...
+
+// convert the Table into a DataSet of Row by specifying a class
+DataSet<Row> dsRow = tableEnv.toDataSet(table, Row.class);
+
+// convert the Table into a DataSet of Tuple2<String, Integer> via a TypeInformation
+TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(
+ Types.STRING(),
+ Types.INT());
+DataSet<Tuple2<String, Integer>> dsTuple =
+ tableEnv.toDataSet(table, tupleType);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get TableEnvironment
+// registration of a DataSet is equivalent
+val tableEnv = BatchTableEnvironment.create(env)
+
+// Table with two fields (String name, Integer age)
+val table: Table = ...
+
+// convert the Table into a DataSet of Row
+val dsRow: DataSet[Row] = tableEnv.toDataSet[Row](table)
+
+// convert the Table into a DataSet of Tuple2[String, Int]
+val dsTuple: DataSet[(String, Int)] = tableEnv.toDataSet[(String, Int)](table)
+{% endhighlight %}
+</div>
+</div>
+
+<span class="label label-danger">Attention</span> **Once the Table is converted to a DataSet, we must use the ExecutionEnvironment.execute method to execute the DataSet program.**
+
{% top %}
### Mapping of Data Types to Table Schema
-Flink's DataStream API support many diverse types.
-Composite types such as Tuples (built-in Scala and Flink Java tuples), POJOs, Scala case classes, and Flink's Row type allow for nested data structures with multiple fields that can be accessed in table expressions. Other types are treated as atomic types. In the following, we describe how the Table API converts these types into an internal row representation and show examples of converting a `DataStream` into a `Table`.
+Flink's DataStream and DataSet APIs support very diverse types. Composite types such as Tuples (built-in Scala and Flink Java tuples), POJOs, Scala case classes, and Flink's Row type allow for nested data structures with multiple fields that can be accessed in table expressions. Other types are treated as atomic types. In the following, we describe how the Table API converts these types into an internal row representation and show examples of converting a `DataStream` into a `Table`.
The mapping of a data type to a table schema can happen in two ways: **based on the field positions** or **based on the field names**.
@@ -1037,35 +1185,35 @@ val table: Table = tableEnv.fromDataStream(stream, $"_2" as "myInt", $"_1" as "m
#### Atomic Types
-Flink treats primitives (`Integer`, `Double`, `String`) or generic types (types that cannot be analyzed and decomposed) as atomic types.
-A `DataStream` of an atomic type is converted into a `Table` with a single column.
-The type of the column is inferred from the atomic type and the name of the column can be specified.
+Flink treats primitives (`Integer`, `Double`, `String`) or generic types (types that cannot be analyzed and decomposed) as atomic types. A `DataStream` or `DataSet` of an atomic type is converted into a `Table` with a single attribute. The type of the attribute is inferred from the atomic type and the name of the attribute can be specified.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-StreamTableEnvironment tableEnv = ...;
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
DataStream<Long> stream = ...
-// Convert DataStream into Table with default field name "f0"
+// convert DataStream into Table with default field name "f0"
Table table = tableEnv.fromDataStream(stream);
-// Convert DataStream into Table with field name "myLong"
+// convert DataStream into Table with field name "myLong"
Table table = tableEnv.fromDataStream(stream, $("myLong"));
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val tableEnv: StreamTableEnvironment = ???
+// get a TableEnvironment
+val tableEnv: StreamTableEnvironment = ... // see "Create a TableEnvironment" section
val stream: DataStream[Long] = ...
-// Convert DataStream into Table with default field name "f0"
+// convert DataStream into Table with default field name "f0"
val table: Table = tableEnv.fromDataStream(stream)
-// Convert DataStream into Table with field name "myLong"
+// convert DataStream into Table with field name "myLong"
val table: Table = tableEnv.fromDataStream(stream, $"myLong")
{% endhighlight %}
</div>
@@ -1073,12 +1221,7 @@ val table: Table = tableEnv.fromDataStream(stream, $"myLong")
#### Tuples (Scala and Java) and Case Classes (Scala only)
-Flink supports Scala's built-in tuples and provides its own tuple classes for Java.
-DataStreams of both kinds of tuples can be converted into tables.
-Fields can be renamed by providing names for all fields (mapping based on position).
-If no field names are specified, the default field names are used.
-If the original field names (`f0`, `f1`, ... for Flink Tuples and `_1`, `_2`, ... for Scala Tuples) are referenced, the API assumes that the mapping is name-based instead of position-based.
-Name-based mapping allows for reordering fields and projection with alias (`as`).
+Flink supports Scala's built-in tuples and provides its own tuple classes for Java. DataStreams and DataSets of both kinds of tuples can be converted into tables. Fields can be renamed by providing names for all fields (mapping based on position). If no field names are specified, the default field names are used. If the original field names (`f0`, `f1`, ... for Flink Tuples and `_1`, `_2`, ... for Scala Tuples) are referenced, the API assumes that the mapping is name-based instead of pos [...]
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -1148,7 +1291,7 @@ val table: Table = tableEnv.fromDataStream(stream, $"age" as "myAge", $"name" as
Flink supports POJOs as composite types. The rules for what determines a POJO are documented [here]({% link dev/types_serialization.md %}#pojos).
-When converting a POJO `DataStream` into a `Table` without specifying field names, the names of the original POJO fields are used. The name mapping requires the original names and cannot be done by positions. Fields can be renamed using an alias (with the `as` keyword), reordered, and projected.
+When converting a POJO `DataStream` or `DataSet` into a `Table` without specifying field names, the names of the original POJO fields are used. The name mapping requires the original names and cannot be done by positions. Fields can be renamed using an alias (with the `as` keyword), reordered, and projected.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -1198,54 +1341,55 @@ val table: Table = tableEnv.fromDataStream(stream, $"name" as "myName")
#### Row
-The `Row` data type supports an arbitrary number of fields and fields with `null` values. Field names can be specified via a `RowTypeInfo` or when converting a `Row` `DataStream` into a `Table`.
-The row type supports mapping of fields by position and by name.
-Fields can be renamed by providing names for all fields (mapping based on position) or selected individually for projection/ordering/renaming (mapping based on name).
+The `Row` data type supports an arbitrary number of fields and fields with `null` values. Field names can be specified via a `RowTypeInfo` or when converting a `Row` `DataStream` or `DataSet` into a `Table`. The row type supports mapping of fields by position and by name. Fields can be renamed by providing names for all fields (mapping based on position) or selected individually for projection/ordering/renaming (mapping based on name).
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-StreamTableEnvironment tableEnv = ...;
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section
// DataStream of Row with two fields "name" and "age" specified in `RowTypeInfo`
DataStream<Row> stream = ...
-// Convert DataStream into Table with default field names "name", "age"
+// convert DataStream into Table with default field names "name", "age"
Table table = tableEnv.fromDataStream(stream);
-// Convert DataStream into Table with renamed field names "myName", "myAge" (position-based)
+// convert DataStream into Table with renamed field names "myName", "myAge" (position-based)
Table table = tableEnv.fromDataStream(stream, $("myName"), $("myAge"));
-// Convert DataStream into Table with renamed fields "myName", "myAge" (name-based)
+// convert DataStream into Table with renamed fields "myName", "myAge" (name-based)
Table table = tableEnv.fromDataStream(stream, $("name").as("myName"), $("age").as("myAge"));
-// Convert DataStream into Table with projected field "name" (name-based)
+// convert DataStream into Table with projected field "name" (name-based)
Table table = tableEnv.fromDataStream(stream, $("name"));
-// Convert DataStream into Table with projected and renamed field "myName" (name-based)
+// convert DataStream into Table with projected and renamed field "myName" (name-based)
Table table = tableEnv.fromDataStream(stream, $("name").as("myName"));
{% endhighlight %}
</div>
+
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val tableEnv: StreamTableEnvironment = ???
+// get a TableEnvironment
+val tableEnv: StreamTableEnvironment = ... // see "Create a TableEnvironment" section
// DataStream of Row with two fields "name" and "age" specified in `RowTypeInfo`
val stream: DataStream[Row] = ...
-// Convert DataStream into Table with default field names "name", "age"
+// convert DataStream into Table with default field names "name", "age"
val table: Table = tableEnv.fromDataStream(stream)
-// Convert DataStream into Table with renamed field names "myName", "myAge" (position-based)
+// convert DataStream into Table with renamed field names "myName", "myAge" (position-based)
val table: Table = tableEnv.fromDataStream(stream, $"myName", $"myAge")
-// Convert DataStream into Table with renamed fields "myName", "myAge" (name-based)
+// convert DataStream into Table with renamed fields "myName", "myAge" (name-based)
val table: Table = tableEnv.fromDataStream(stream, $"name" as "myName", $"age" as "myAge")
-// Convert DataStream into Table with projected field "name" (name-based)
+// convert DataStream into Table with projected field "name" (name-based)
val table: Table = tableEnv.fromDataStream(stream, $"name")
-// Convert DataStream into Table with projected and renamed field "myName" (name-based)
+// convert DataStream into Table with projected and renamed field "myName" (name-based)
val table: Table = tableEnv.fromDataStream(stream, $"name" as "myName")
{% endhighlight %}
</div>
@@ -1257,6 +1401,8 @@ val table: Table = tableEnv.fromDataStream(stream, $"name" as "myName")
Query Optimization
------------------
+<div class="codetabs" markdown="1">
+<div data-lang="Blink planner" markdown="1">
Apache Flink leverages and extends Apache Calcite to perform sophisticated query optimization.
This includes a series of rule and cost-based optimizations such as:
@@ -1276,6 +1422,14 @@ This includes a series of rule and cost-based optimizations such as:
The optimizer makes intelligent decisions, based not only on the plan but also rich statistics available from the data sources and fine-grain costs for each operator such as io, cpu, network, and memory.
Advanced users may provide custom optimizations via a `CalciteConfig` object that can be provided to the table environment by calling `TableEnvironment#getConfig#setPlannerConfig`.
+</div>
+
+<div data-lang="Old planner" markdown="1">
+Apache Flink leverages Apache Calcite to optimize and translate queries. The optimization currently performed include projection and filter push-down, subquery decorrelation, and other kinds of query rewriting. Old planner does not yet optimize the order of joins, but executes them in the same order as defined in the query (order of Tables in the `FROM` clause and/or order of join predicates in the `WHERE` clause).
+
+It is possible to tweak the set of optimization rules which are applied in different phases by providing a `CalciteConfig` object. This can be created via a builder by calling `CalciteConfig.createBuilder())` and is provided to the TableEnvironment by calling `tableEnv.getConfig.setPlannerConfig(calciteConfig)`.
+</div>
+</div>
Explaining a Table
diff --git a/docs/dev/table/index.md b/docs/dev/table/index.md
index 94d8a00..9b726ea 100644
--- a/docs/dev/table/index.md
+++ b/docs/dev/table/index.md
@@ -83,8 +83,10 @@ $ python -m pip install apache-flink
</div>
Additionally, if you want to run the Table API & SQL programs locally within your IDE, you must add the
-following set of modules.
+following set of modules, depending which planner you want to use.
+<div class="codetabs" markdown="1">
+<div data-lang="Blink Planner" markdown="1">
{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
@@ -99,6 +101,24 @@ following set of modules.
<scope>provided</scope>
</dependency>
{% endhighlight %}
+</div>
+<div data-lang="Legacy Planner" markdown="1">
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
+ <scope>provided</scope>
+</dependency>
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
+ <scope>provided</scope>
+</dependency>
+{% endhighlight %}
+</div>
+</div>
### Extension Dependencies
diff --git a/docs/dev/table/legacy_planner.md b/docs/dev/table/legacy_planner.md
deleted file mode 100644
index 5f4e824..0000000
--- a/docs/dev/table/legacy_planner.md
+++ /dev/null
@@ -1,274 +0,0 @@
----
-title: "Legacy Planner"
-nav-parent_id: tableapi
-nav-pos: 1001
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Table planners are responsible for translating relational operators into an executable, optimized Flink job.
-Flink supports two different planner implementations; the modern planner (sometimes referred to as `Blink`) and the legacy planner.
-For production use cases, we recommend the modern planner which is the default.
-
-The legacy planner is in maintenance mode and no longer under active development.
-The primary reason to continue using the legacy planner is [DataSet]({% link dev/batch/index.md %}) interop.
-
-This page describes how to use the Legacy planner and where its semantics differ from the
-modern planner.
-
-* This will be replaced by the TOC
-{:toc}
-
-## Setup
-
-### Dependencies
-
-When deploying to a cluster, the legacy planner is bundled in Flinks distribution by default.
-If you want to run the Table API & SQL programs locally within your IDE, you must add the
-following set of modules to your application.
-
-{% highlight xml %}
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version}}</version>
- <scope>provided</scope>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version}}</version>
- <scope>provided</scope>
-</dependency>
-{% endhighlight %}
-
-### Configuring the TableEnvironment
-
-When creating a `TableEnvironment` the Legacy planner is configured via the `EnvironmentSettings`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-EnvironmentSettings settings = EnvironmentSettings
- .newInstance()
- .useOldPlanner()
- .inStreamingMode()
- //.inBatchMode()
- .build();
-
-TableEnvironment tEnv = TableEnvironment.create(settings);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val settings = EnvironmentSettings
- .newInstance()
- .useOldPlanner()
- .inStreamingMode()
- //.inBatchMode()
- .build()
-
-val tEnv = TableEnvironment.create(settings)
-{% endhighlight %}
-</div>
-</div>
-
-`BatchTableEnvironment` may used for [DataSet]({% link dev/batch/index.md %}) and [DataStream]({% link dev/datastream_api.md %}) interop respectively.
-
-{% capture custom_operator_note %}
-If you are not using the Legacy planner for DataSet interop, the community strongly
-encourages you to consider the modern table planner.
-{% endcapture %}
-{% include note.html content=custom_operator_note %}
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tEnv = BatchTableEnvironment.create(env);
-
-EnvironmentSettings fsSettings = EnvironmentSettings
- .newInstance()
- .useOldPlanner()
- .inStreamingMode()
- .build();
-
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tEnv = StreamTableEnvironment.create(fsEnv, fsSettings);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment()
-val tEnv = BatchTableEnvironment.create(env)
-
-val fsSettings = EnvironmentSettings
- .newInstance()
- .useOldPlanner()
- .inStreamingMode()
- .build()
-
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment()
-StreamTableEnvironment tEnv = StreamTableEnvironment.create(fsEnv, fsSettings)
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-from pyflink.dataset import ExecutionEnvironment
-from pyflink.table import BatchTableEnvironment
-
-f_b_env = ExecutionEnvironment.get_execution_environment()
-f_b_t_env = BatchTableEnvironment.create(f_b_env, table_config)
-
-from pyflink.datastream import StreamExecutionEnvironment
-from pyflink.table import StreamTableEnvironment, EnvironmentSettings
-
-f_s_env = StreamExecutionEnvironment.get_execution_environment()
-f_s_settings = EnvironmentSettings.new_instance().use_old_planner().in_streaming_mode().build()
-f_s_t_env = StreamTableEnvironment.create(f_s_env, environment_settings=f_s_settings)
-{% endhighlight %}
-</div>
-</div>
-
-
-## Integration with DataSet
-
-The primary use case for the Legacy planner is interoperation with the DataSet API.
-To translate `DataSet`s to and from tables, applications must use the `BatchTableEnvironment`.
-
-### Create a View from a DataSet
-
-A `DataSet` can be registered in a `BatchTableEnvironment` as a `View`.
-The schema of the resulting view depends on the data type of the registered collection.
-
-**Note:** Views created from a `DataSet` can be registered as temporary views only.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-BatchTableEnvironment tEnv = ...;
-DataSet<Tuple2<Long, String>> dataset = ...;
-
-tEnv.createTemporaryView("my-table", dataset, $("myLong"), $("myString"))
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight java %}
-val tEnv: BatchTableEnvironment = ???
-val dataset: DataSet[(Long, String)] = ???
-
-tEnv.createTemporaryView("my-table", dataset, $"myLong", $"myString")
-{% endhighlight %}
-</div>
-</div>
-
-### Create a Table from a DataSet
-
-A `DataSet` can be directly converted to a `Table` in a `BatchTableEnvironment`.
-The schema of the resulting view depends on the data type of the registered collection.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-BatchTableEnvironment tEnv = ...;
-DataSet<Tuple2<Long, String>> dataset = ...;
-
-Table myTable = tEnv.fromDataSet("my-table", dataset, $("myLong"), $("myString"))
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight java %}
-val tEnv: BatchTableEnvironment = ???
-val dataset: DataSet[(Long, String)] = ???
-
-val table = tEnv.fromDataSet("my-table", dataset, $"myLong", $"myString")
-{% endhighlight %}
-</div>
-</div>
-
-### Convert a Table to a DataSet
-
-A `Table` can be converted to a `DataSet`.
-In this way, custom DataSet programs can be run on the result of a Table API or SQL query.
-
-When converting from a `Table`, users must specify the data type of the results.
-Often the most convenient conversion type is `Row`.
-The following list gives an overview of the features of the different options.
-
-- **Row**: fields are mapped by position, arbitrary number of fields, support for `null` values, no type-safe access.
-- **POJO**: fields are mapped by name (POJO fields must be named as `Table` fields), arbitrary number of fields, support for `null` values, type-safe access.
-- **Case Class**: fields are mapped by position, no support for `null` values, type-safe access.
-- **Tuple**: fields are mapped by position, limitation to 22 (Scala) or 25 (Java) fields, no support for `null` values, type-safe access.
-- **Atomic Type**: `Table` must have a single field, no support for `null` values, type-safe access.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env);
-
-Table table = tableEnv.fromValues(
- DataTypes.Row(
- DataTypes.FIELD("name", DataTypes.STRING()),
- DataTypes.FIELD("age", DataTypes.INT()),
- row("john", 35),
- row("sarah", 32));
-
-// Convert the Table into a DataSet of Row by specifying a class
-DataSet<Row> dsRow = tableEnv.toDataSet(table, Row.class);
-
-// Convert the Table into a DataSet of Tuple2<String, Integer> via a TypeInformation
-TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(Types.STRING(), Types.INT());
-DataSet<Tuple2<String, Integer>> dsTuple = tableEnv.toDataSet(table, tupleType);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val tableEnv = BatchTableEnvironment.create(env)
-
-val table = tableEnv.fromValues(
- DataTypes.Row(
- DataTypes.FIELD("name", DataTypes.STRING()),
- DataTypes.FIELD("age", DataTypes.INT()),
- row("john", 35),
- row("sarah", 32));
-
-// Convert the Table into a DataSet of Row
-val dsRow: DataSet[Row] = tableEnv.toDataSet[Row](table)
-
-// Convert the Table into a DataSet of Tuple2[String, Int]
-val dsTuple: DataSet[(String, Int)] = tableEnv.toDataSet[(String, Int)](table)
-{% endhighlight %}
-</div>
-</div>
-
-<span class="label label-danger">Attention</span> **Once the Table is converted to a DataSet, we must use the ExecutionEnvironment.execute method to execute the DataSet program.**
-
-{% top %}
-
-Main Differences Between the Two Planners
------------------------------------------
-
-1. Blink treats batch jobs as a special case of streaming. As such, the conversion between Table and DataSet is also not supported, and batch jobs will not be translated into `DateSet` programs but translated into `DataStream` programs, the same as the streaming jobs.
-2. The Blink planner does not support `BatchTableSource`, use bounded `StreamTableSource` instead of it.
-3. The implementations of `FilterableTableSource` for the old planner and the Blink planner are incompatible. The old planner will push down `PlannerExpression`s into `FilterableTableSource`, while the Blink planner will push down `Expression`s.
-4. String based key-value config options (Please see the documentation about [Configuration]({% link dev/table/config.md %}) for details) are only used for the Blink planner.
-5. The implementation(`CalciteConfig`) of `PlannerConfig` in two planners is different.
-6. The Blink planner will optimize multiple-sinks into one DAG on both `TableEnvironment` and `StreamTableEnvironment`. The old planner will always optimize each sink into a new DAG, where all DAGs are independent of each other.
-7. The old planner does not support catalog statistics now, while the Blink planner does.
-
diff --git a/docs/redirects/temporal_table.md b/docs/redirects/temporal_table.md
index 9e3bfa1..a7c1461 100644
--- a/docs/redirects/temporal_table.md
+++ b/docs/redirects/temporal_table.md
@@ -1,7 +1,7 @@
---
title: Temporal Tables
layout: redirect
-redirect: /dev/table/streaming/legacy.html
+redirect: /dev/table/streaming/versioned_table.html
permalink: /dev/table/streaming/temporal_tables.html
---
<!--