You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/12 17:12:51 UTC
[GitHub] [flink] sjwiesman opened a new pull request #14053: [FLINK-18279][docs] Simplify Table Overview
sjwiesman opened a new pull request #14053:
URL: https://github.com/apache/flink/pull/14053
## What is the purpose of the change
The table overview page contains an overwhelming amount of information. We should simplify the page so users quickly know:
* What dependencies they actually need to add in their user code to begin developing
* Which planner to use
* Updated links to the modern link tag
Relevant portion rendered
<img width="310" alt="Screen Shot 2020-11-12 at 11 12 08 AM" src="https://user-images.githubusercontent.com/1891970/98972353-f14f3280-24d7-11eb-8fec-ce00382caa7a.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-726229826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60731cdca7d268d5a98103ebdb81867a0013bf11 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #14053:
URL: https://github.com/apache/flink/pull/14053#discussion_r528738848
##########
File path: docs/dev/table/index.md
##########
@@ -25,93 +25,106 @@ specific language governing permissions and limitations
under the License.
-->
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Dependency Structure
---------------------
-
-Starting from Flink 1.9, Flink provides two different planner implementations for evaluating Table & SQL API programs: the Blink planner and the old planner that was available before Flink 1.9. Planners are responsible for
-translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes.
-They may also differ in the set of supported features.
-
-<span class="label label-danger">Attention</span> For production use cases, we recommend the blink planner that has become the default planner since 1.11.
-
-All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts.
-
-The following dependencies are relevant for most projects:
-
-* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
-* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
-* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
-* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
-* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
-* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11.
-* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11.
-* `flink-table-runtime-blink`: The new Blink runtime.
-* `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default.
-* `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default.
-
-See the [common API](common.html) page for more information about how to switch between the old and new Blink planner in table programs.
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch
+processing. The Table API is a language-integrated query API for Java, Scala, and Python that
+allows the composition of queries from relational operators such as selection, filter, and join in
+a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org)
+which implements the SQL standard. Queries specified in either interface have the same semantics
+and specify the same result regardless of whether the input is continuous (streaming) or bounded (batch).
+
+The Table API and SQL interfaces integrate seamlessly with each other and Flink's DataStream API.
+You can easily switch between all APIs and libraries which build upon them.
+For instance, you can extract patterns from a Table using [Match Recognize]({% link dev/table/streaming/match_recognize.md %})
+and later use the DataStream API to build alerting based on the matched patterns.
+
+Table Planners
+--------------
+
+Table planners are responsible for translating relational operators into an executable, optimized Flink job.
+Flink supports two different planner implementations; the modern Blink planner and the legacy planner.
+For production use cases, we recommend the blink planner which has been the default planner since 1.11.
Review comment:
```suggestion
For production use cases, we recommend the Blink planner which has been the default planner since 1.11.
```
##########
File path: docs/dev/table/index.md
##########
@@ -25,93 +25,106 @@ specific language governing permissions and limitations
under the License.
-->
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Dependency Structure
---------------------
-
-Starting from Flink 1.9, Flink provides two different planner implementations for evaluating Table & SQL API programs: the Blink planner and the old planner that was available before Flink 1.9. Planners are responsible for
-translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes.
-They may also differ in the set of supported features.
-
-<span class="label label-danger">Attention</span> For production use cases, we recommend the blink planner that has become the default planner since 1.11.
-
-All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts.
-
-The following dependencies are relevant for most projects:
-
-* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
-* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
-* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
-* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
-* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
-* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11.
-* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11.
-* `flink-table-runtime-blink`: The new Blink runtime.
-* `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default.
-* `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default.
-
-See the [common API](common.html) page for more information about how to switch between the old and new Blink planner in table programs.
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch
+processing. The Table API is a language-integrated query API for Java, Scala, and Python that
+allows the composition of queries from relational operators such as selection, filter, and join in
+a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org)
+which implements the SQL standard. Queries specified in either interface have the same semantics
+and specify the same result regardless of whether the input is continuous (streaming) or bounded (batch).
+
+The Table API and SQL interfaces integrate seamlessly with each other and Flink's DataStream API.
+You can easily switch between all APIs and libraries which build upon them.
+For instance, you can extract patterns from a Table using [Match Recognize]({% link dev/table/streaming/match_recognize.md %})
Review comment:
```suggestion
For instance, you can detect patterns from a table using [`MATCH_RECOGNIZE` clause]({% link dev/table/streaming/match_recognize.md %})
```
##########
File path: docs/dev/table/index.md
##########
@@ -25,93 +25,106 @@ specific language governing permissions and limitations
under the License.
-->
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Dependency Structure
---------------------
-
-Starting from Flink 1.9, Flink provides two different planner implementations for evaluating Table & SQL API programs: the Blink planner and the old planner that was available before Flink 1.9. Planners are responsible for
-translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes.
-They may also differ in the set of supported features.
-
-<span class="label label-danger">Attention</span> For production use cases, we recommend the blink planner that has become the default planner since 1.11.
-
-All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts.
-
-The following dependencies are relevant for most projects:
-
-* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
-* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
-* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
-* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
-* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
-* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11.
-* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11.
-* `flink-table-runtime-blink`: The new Blink runtime.
-* `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default.
-* `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default.
-
-See the [common API](common.html) page for more information about how to switch between the old and new Blink planner in table programs.
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch
+processing. The Table API is a language-integrated query API for Java, Scala, and Python that
+allows the composition of queries from relational operators such as selection, filter, and join in
+a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org)
+which implements the SQL standard. Queries specified in either interface have the same semantics
+and specify the same result regardless of whether the input is continuous (streaming) or bounded (batch).
+
+The Table API and SQL interfaces integrate seamlessly with each other and Flink's DataStream API.
+You can easily switch between all APIs and libraries which build upon them.
+For instance, you can extract patterns from a Table using [Match Recognize]({% link dev/table/streaming/match_recognize.md %})
+and later use the DataStream API to build alerting based on the matched patterns.
+
+Table Planners
+--------------
+
+Table planners are responsible for translating relational operators into an executable, optimized Flink job.
+Flink supports two different planner implementations; the modern Blink planner and the legacy planner.
+For production use cases, we recommend the blink planner which has been the default planner since 1.11.
+See the [common API]({% link dev/table/common.md %}) page for more information on how to switch between the two planners.
### Table Program Dependencies
-Depending on the target programming language, you need to add the Java or Scala API to a project in order to use the Table API & SQL for defining pipelines:
+Depending on the target programming language, you need to add the Java or Scala API to a project
Review comment:
mention Python here as well?
##########
File path: docs/dev/table/index.md
##########
@@ -25,93 +25,106 @@ specific language governing permissions and limitations
under the License.
-->
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Dependency Structure
---------------------
-
-Starting from Flink 1.9, Flink provides two different planner implementations for evaluating Table & SQL API programs: the Blink planner and the old planner that was available before Flink 1.9. Planners are responsible for
-translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes.
-They may also differ in the set of supported features.
-
-<span class="label label-danger">Attention</span> For production use cases, we recommend the blink planner that has become the default planner since 1.11.
-
-All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts.
-
-The following dependencies are relevant for most projects:
-
-* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
-* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
-* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
-* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
-* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
-* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11.
-* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11.
-* `flink-table-runtime-blink`: The new Blink runtime.
-* `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default.
-* `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default.
-
-See the [common API](common.html) page for more information about how to switch between the old and new Blink planner in table programs.
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch
+processing. The Table API is a language-integrated query API for Java, Scala, and Python that
+allows the composition of queries from relational operators such as selection, filter, and join in
+a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org)
+which implements the SQL standard. Queries specified in either interface have the same semantics
+and specify the same result regardless of whether the input is continuous (streaming) or bounded (batch).
+
+The Table API and SQL interfaces integrate seamlessly with each other and Flink's DataStream API.
+You can easily switch between all APIs and libraries which build upon them.
+For instance, you can extract patterns from a Table using [Match Recognize]({% link dev/table/streaming/match_recognize.md %})
+and later use the DataStream API to build alerting based on the matched patterns.
+
+Table Planners
+--------------
+
+Table planners are responsible for translating relational operators into an executable, optimized Flink job.
+Flink supports two different planner implementations; the modern Blink planner and the legacy planner.
+For production use cases, we recommend the blink planner which has been the default planner since 1.11.
+See the [common API]({% link dev/table/common.md %}) page for more information on how to switch between the two planners.
### Table Program Dependencies
-Depending on the target programming language, you need to add the Java or Scala API to a project in order to use the Table API & SQL for defining pipelines:
+Depending on the target programming language, you need to add the Java or Scala API to a project
+in order to use the Table API & SQL for defining pipelines.
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
{% highlight xml %}
-<!-- Either... -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
-<!-- or... -->
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-scala-bridge{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
+</div>
+<div data-lang="python">
+{% highlight bash %}
+{% if site.is_stable %}
+$ python -m pip install apache-flink {{ site.version }}
+{% else %}
+$ python -m pip install apache-flink
+{% endif %}
+{% endhighlight %}
+</div>
+</div>
-Additionally, if you want to run the Table API & SQL programs locally within your IDE, you must add one of the
-following set of modules, depending which planner you want to use:
+Additionally, if you want to run the Table API & SQL programs locally within your IDE, you must add the
+following set of modules, depending which planner you want to use.
+<div class="codetabs" markdown="1">
+<div data-lang="Blink Planner" markdown="1">
{% highlight xml %}
-<!-- Either... (for the old planner that was available before Flink 1.9) -->
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <artifactId>flink-table-planner-blink{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
-<!-- or.. (for the new Blink planner) -->
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner-blink{{ site.scala_version_suffix }}</artifactId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
-
-Internally, parts of the table ecosystem are implemented in Scala. Therefore, please make sure to add the following dependency for both batch and streaming applications:
-
+</div>
+<div data-lang="Legacy Planner" markdown="1">
{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
+ <scope>provided</scope>
+</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
+</div>
+</div>
### Extension Dependencies
-If you want to implement a [custom format]({{ site.baseurl }}/dev/table/sourceSinks.html#define-a-tablefactory) for interacting with Kafka or a set of [user-defined functions]({{ site.baseurl }}/dev/table/functions/systemFunctions.html), the following dependency is sufficient and can be used for JAR files for the SQL Client:
+If you want to implement a [custom format]({% link dev/table/sourceSinks.md %}#define-a-tablefactory)
+for (de)serializing rows or a set of [user-defined functions]({% link dev/table/functions/systemFunctions.md %}),
Review comment:
```suggestion
for (de)serializing rows or a set of [user-defined functions]({% link dev/table/functions/udfs.md %}),
```
##########
File path: docs/dev/table/index.md
##########
@@ -25,93 +25,106 @@ specific language governing permissions and limitations
under the License.
-->
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Dependency Structure
---------------------
-
-Starting from Flink 1.9, Flink provides two different planner implementations for evaluating Table & SQL API programs: the Blink planner and the old planner that was available before Flink 1.9. Planners are responsible for
-translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes.
-They may also differ in the set of supported features.
-
-<span class="label label-danger">Attention</span> For production use cases, we recommend the blink planner that has become the default planner since 1.11.
-
-All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts.
-
-The following dependencies are relevant for most projects:
-
-* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
-* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
-* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
-* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
-* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
-* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11.
-* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11.
-* `flink-table-runtime-blink`: The new Blink runtime.
-* `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default.
-* `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default.
-
-See the [common API](common.html) page for more information about how to switch between the old and new Blink planner in table programs.
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch
+processing. The Table API is a language-integrated query API for Java, Scala, and Python that
+allows the composition of queries from relational operators such as selection, filter, and join in
+a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org)
+which implements the SQL standard. Queries specified in either interface have the same semantics
+and specify the same result regardless of whether the input is continuous (streaming) or bounded (batch).
+
+The Table API and SQL interfaces integrate seamlessly with each other and Flink's DataStream API.
+You can easily switch between all APIs and libraries which build upon them.
+For instance, you can extract patterns from a Table using [Match Recognize]({% link dev/table/streaming/match_recognize.md %})
+and later use the DataStream API to build alerting based on the matched patterns.
+
+Table Planners
+--------------
+
+Table planners are responsible for translating relational operators into an executable, optimized Flink job.
+Flink supports two different planner implementations; the modern Blink planner and the legacy planner.
+For production use cases, we recommend the blink planner which has been the default planner since 1.11.
+See the [common API]({% link dev/table/common.md %}) page for more information on how to switch between the two planners.
### Table Program Dependencies
-Depending on the target programming language, you need to add the Java or Scala API to a project in order to use the Table API & SQL for defining pipelines:
+Depending on the target programming language, you need to add the Java or Scala API to a project
+in order to use the Table API & SQL for defining pipelines.
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
{% highlight xml %}
-<!-- Either... -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
-<!-- or... -->
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-scala-bridge{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
+</div>
+<div data-lang="python">
+{% highlight bash %}
+{% if site.is_stable %}
+$ python -m pip install apache-flink {{ site.version }}
+{% else %}
+$ python -m pip install apache-flink
+{% endif %}
+{% endhighlight %}
+</div>
+</div>
-Additionally, if you want to run the Table API & SQL programs locally within your IDE, you must add one of the
-following set of modules, depending which planner you want to use:
+Additionally, if you want to run the Table API & SQL programs locally within your IDE, you must add the
+following set of modules, depending which planner you want to use.
+<div class="codetabs" markdown="1">
+<div data-lang="Blink Planner" markdown="1">
{% highlight xml %}
-<!-- Either... (for the old planner that was available before Flink 1.9) -->
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <artifactId>flink-table-planner-blink{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
-<!-- or.. (for the new Blink planner) -->
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-table-planner-blink{{ site.scala_version_suffix }}</artifactId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
-
-Internally, parts of the table ecosystem are implemented in Scala. Therefore, please make sure to add the following dependency for both batch and streaming applications:
-
+</div>
+<div data-lang="Legacy Planner" markdown="1">
{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
+ <scope>provided</scope>
+</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
<scope>provided</scope>
</dependency>
{% endhighlight %}
+</div>
+</div>
### Extension Dependencies
-If you want to implement a [custom format]({{ site.baseurl }}/dev/table/sourceSinks.html#define-a-tablefactory) for interacting with Kafka or a set of [user-defined functions]({{ site.baseurl }}/dev/table/functions/systemFunctions.html), the following dependency is sufficient and can be used for JAR files for the SQL Client:
+If you want to implement a [custom format]({% link dev/table/sourceSinks.md %}#define-a-tablefactory)
Review comment:
```suggestion
If you want to implement a [custom format or connector]({% link dev/table/sourceSinks.md %})
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-726214159
@morsapaes @twalthr Would either of you like to take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-726215589
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 60731cdca7d268d5a98103ebdb81867a0013bf11 (Thu Nov 12 17:15:26 UTC 2020)
**Warnings:**
* Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-732230487
Thanks for the review @twalthr. Since the `zh` page hasn't been translated yet I was just going to wait to sync until merging so I don't have to coordinate changes across pages during review :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman edited a comment on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
sjwiesman edited a comment on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-732230487
Thanks for the review @twalthr. Since the `zh` page has been translated I was just going to open a follow up ticket. Will merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-726229826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9529",
"triggerID" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60731cdca7d268d5a98103ebdb81867a0013bf11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9529)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14053:
URL: https://github.com/apache/flink/pull/14053#issuecomment-726229826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9529",
"triggerID" : "60731cdca7d268d5a98103ebdb81867a0013bf11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60731cdca7d268d5a98103ebdb81867a0013bf11 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9529)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #14053: [FLINK-18279][docs] Simplify Table Overview
Posted by GitBox <gi...@apache.org>.
sjwiesman closed pull request #14053:
URL: https://github.com/apache/flink/pull/14053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org