You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/06/19 14:06:39 UTC

[GitHub] [flink] twalthr opened a new pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

twalthr opened a new pull request #12724:
URL: https://github.com/apache/flink/pull/12724


   ## What is the purpose of the change
   
   Adds documentation and examples for the new table sources and sinks architecture. This is just initial documentation. `LookupTableSource` and ability interfaces are still missing.
   
   ## Brief change log
   
   See commit messages.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? docs
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7776887183ee01b9b7f05ef4d3afeb892679faff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835) 
   * 45594051f1293ba9d2309188224e055193ca4387 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] aljoscha commented on a change in pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #12724:
URL: https://github.com/apache/flink/pull/12724#discussion_r443508462



##########
File path: flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketDynamicTableSource.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java.connectors;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.ScanTableSource;
+import org.apache.flink.table.connector.source.SourceFunctionProvider;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+
+/**
+ * The {@link SocketDynamicTableSource} is used during planning.
+ *
+ * <p>In our example, we don't implement any of the available ability interfaces. Therefore, the main

Review comment:
       Here the reader will not know what "ability interfaces" are.

##########
File path: flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketSourceFunction.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java.connectors;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
+import org.apache.flink.table.data.RowData;
+
+import java.io.ByteArrayOutputStream;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+
+/**
+ * The {@link SocketSourceFunction} opens a socket and consumes bytes.
+ *
+ * <p>It splits records by the given byte delimiter (`\n` by default) and delegates the decoding to a
+ * pluggable {@link DeserializationSchema}.
+ *
+ * <p>Note: The source function can only work with a parallelism of 1.

Review comment:
       Maybe also say that it's not fault-tolerant and shouldn't be used in production.

##########
File path: flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/ChangelogSocketExample.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.examples.java.connectors;
+
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.types.Row;
+
+/**
+ * Example for implementing a custom {@link DynamicTableSource} and a {@link DecodingFormat}.
+ *
+ * <p>The example implements a table source with a decoding format that supports changelog semantics.
+ *
+ * <p>The {@link SocketDynamicTableFactory} illustrates how connector components play together. It can
+ * serve as a reference implementation for implementing own connectors and/or formats.
+ *
+ * <p>The {@link SocketDynamicTableSource} uses a simple single-threaded {@link SourceFunction} to open
+ * a socket that listens for incoming bytes. The raw bytes are decoded into rows by a pluggable format.
+ * The format expects a changelog flag as the first column.
+ *
+ * <p>In particular, the example shows how to
+ * <ul>
+ *     <li>create factories that parse and validate options,
+ *     <li>implement table connectors,
+ *     <li>implement and discover custom formats,
+ *     <li>and use provided utilities such as data structure converters and the {@link FactoryUtil}.
+ * </ul>
+ *
+ * <p>Usage: <code>ChangelogSocketExample --hostname &lt;localhost&gt; --port &lt;9999&gt;</code>
+ *
+ * <p>Use the following command to ingest data in a terminal:
+ * <pre>
+ *     nc -lk 9999
+ *     INSERT|Alice|12
+ *     INSERT|Bob|5
+ *     DELETE|Alice|12
+ *     INSERT|Alice|18
+ * </pre>
+ *
+ * <p>The result is written to stdout.
+ */
+public final class ChangelogSocketExample {
+
+	// *************************************************************************

Review comment:
       I don't think this `PROGRAM` comment is necessary here since we only have a `main()` function.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7776887183ee01b9b7f05ef4d3afeb892679faff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
twalthr commented on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-647612078


   Thanks @aljoscha. I will merge this once I have a green build.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7776887183ee01b9b7f05ef4d3afeb892679faff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835) 
   * 45594051f1293ba9d2309188224e055193ca4387 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] aljoscha commented on a change in pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #12724:
URL: https://github.com/apache/flink/pull/12724#discussion_r443514976



##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared

Review comment:
       ```suggestion
   operation. Connector-specific dependencies don't have to be present in the classpath yet. The options declared
   ```
   _must_ says that it would be an error if they are present.

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime

Review comment:
       I would leave out that they are unified. This text will become outdated and we will forget to change it.

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.

Review comment:
       ```suggestion
   create specialized connectors.
   ```

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog

Review comment:
       ```suggestion
   Dynamic table factories are used to configure a dynamic table connector for an external storage system from catalog
   ```

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog
+and session information.
+
+`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`.
+
+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`.
+
+By default, the factory is discovered using the value of the `connector` option as the factory identifier
+and Java's Service Provider Interface.
+
+In JAR files, references to new implementations can be added to the service file:
+
+`META-INF/services/org.apache.flink.table.factories.Factory`
+
+The framework will check for a single matching factory that is uniquely identified by factory identifier
+and requested base class (e.g. `DynamicTableSourceFactory`).
+
+The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a
+catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`.
+
+### Dynamic Table Source
+
+By definition, a dynamic table can change over time.
+
+When reading a dynamic table, the content can either be considered as:
+- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog
+  is exhausted. Represented in the `ScanTableSource` interface.
+- A continuously changing or very large external table whose content is usually never read entirely
+  but queried for individual values when necessary. Represented in the `LookupTableSource` interface.
+
+Both interfaces can be implemented at the same time. The planner decides about their usage depending
+on the specified query.
+
+#### Scan Table Source
+
+A `ScanTableSource` scans all rows from an external storage system during runtime.
+
+The scanned rows don't have to contain only insertions but can also contain updates and deletions. Thus,
+the table source can be used to read a (finite or infinite) changelog. The returned _changelog mode_ indicates
+the set of changes that the planner can expect during runtime.
+
+For regular batch scenarios, the source can emit a bounded stream of insert-only rows.
+
+For regular streaming scenarios, the source can emit an unbounded stream of insert-only rows.
+
+For change data capture (CDC) scenarios, the source can emit bounded or unbounded streams with insert,
+update, and delete rows.
+
+A table source can implement further abilitiy interfaces such as `SupportsProjectionPushDown` that might
+mutate an instance during planning. All abilities are listed in the `org.apache.flink.table.connector.source.abilities`
+package and in the documentation of `org.apache.flink.table.connector.source.ScanTableSource`.
+
+The runtime implementation of a `ScanTableSource` must produce internal data structures. Thus, records
+must be emitted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such
+that a source can still work on common data structures and perform a conversion at the end.
+
+#### Lookup Table Source
+
+A `LookupTableSource` looks up rows of an external storage system by one or more keys during runtime.
+
+Compared to `ScanTableSource`, the source must not read the entire table and can lazily fetch individual
+values from a (possibly continuously changing) external table when necessary.
+
+Compared to `ScanTableSource`, a `LookupTableSource` does only support emitting insert-only changes currently.
+
+Further abilities are not supported. See the documentation of `org.apache.flink.table.connector.source.LookupTableSource`
+for more information.
+
+The runtime implementation of a `LookupTableSource` is a `TableFunction` or `AsyncTableFunction`. The function
+will be called with values for the given lookup keys during runtime.
+
+### Dynamic Table Sink
+
+By definition, a dynamic table can change over time.
+
+When writing a dynamic table, the content can either be considered as a changelog (finite or infinite)

Review comment:
       the other half of the _either_ seems to be missing: "the content can either be ... *or* ..."

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in

Review comment:
       in `SELECT` queries *OR* in a `SELECT` query

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog
+and session information.
+
+`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`.
+
+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`.
+
+By default, the factory is discovered using the value of the `connector` option as the factory identifier
+and Java's Service Provider Interface.
+
+In JAR files, references to new implementations can be added to the service file:
+
+`META-INF/services/org.apache.flink.table.factories.Factory`
+
+The framework will check for a single matching factory that is uniquely identified by factory identifier
+and requested base class (e.g. `DynamicTableSourceFactory`).
+
+The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a
+catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`.
+
+### Dynamic Table Source
+
+By definition, a dynamic table can change over time.
+
+When reading a dynamic table, the content can either be considered as:
+- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog
+  is exhausted. Represented in the `ScanTableSource` interface.
+- A continuously changing or very large external table whose content is usually never read entirely
+  but queried for individual values when necessary. Represented in the `LookupTableSource` interface.

Review comment:
       ```suggestion
     but queried for individual values when necessary. This is represented by the `LookupTableSource` interface.
   ```

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog
+and session information.
+
+`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`.
+
+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`.
+
+By default, the factory is discovered using the value of the `connector` option as the factory identifier
+and Java's Service Provider Interface.
+
+In JAR files, references to new implementations can be added to the service file:
+
+`META-INF/services/org.apache.flink.table.factories.Factory`
+
+The framework will check for a single matching factory that is uniquely identified by factory identifier
+and requested base class (e.g. `DynamicTableSourceFactory`).
+
+The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a
+catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`.
+
+### Dynamic Table Source
+
+By definition, a dynamic table can change over time.
+
+When reading a dynamic table, the content can either be considered as:
+- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog
+  is exhausted. Represented in the `ScanTableSource` interface.

Review comment:
       ```suggestion
     is exhausted. This is represented by the `ScanTableSource` interface.
   ```

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog
+and session information.
+
+`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`.
+
+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`.
+
+By default, the factory is discovered using the value of the `connector` option as the factory identifier
+and Java's Service Provider Interface.
+
+In JAR files, references to new implementations can be added to the service file:
+
+`META-INF/services/org.apache.flink.table.factories.Factory`
+
+The framework will check for a single matching factory that is uniquely identified by factory identifier
+and requested base class (e.g. `DynamicTableSourceFactory`).
+
+The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a
+catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`.
+
+### Dynamic Table Source
+
+By definition, a dynamic table can change over time.
+
+When reading a dynamic table, the content can either be considered as:
+- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog
+  is exhausted. Represented in the `ScanTableSource` interface.
+- A continuously changing or very large external table whose content is usually never read entirely
+  but queried for individual values when necessary. Represented in the `LookupTableSource` interface.
+
+Both interfaces can be implemented at the same time. The planner decides about their usage depending

Review comment:
       ```suggestion
   A class can implement both of these interfaces at the same time. The planner decides about their usage depending
   ```

##########
File path: docs/dev/table/sourceSinks.md
##########
@@ -0,0 +1,740 @@
+---
+title: "User-defined Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 130
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded
+data in a unified fashion.
+
+Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content
+of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files.
+
+_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In
+the documentation, sources and sinks are often summarized under the term _connector_.
+
+Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %})
+for more information about built-in table sources and sinks.
+
+This page focuses on how to develop a custom, user-defined connector.
+
+<span class="label label-danger">Attention</span> New table source and table sink interfaces have been
+introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces).
+Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces
+are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look
+at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces
+are still supported for backwards compatibility.
+
+* This will be replaced by the TOC
+{:toc}
+
+Overview
+--------
+
+In many cases, implementers don't need to create a new connector from scratch but would like to slightly
+modify existing connectors or hook into the existing stack. In other cases, implementers would like to
+create specialed connectors.
+
+This section helps for both kinds of use cases. It explains the general architecture of table connectors
+from pure declaration in the API to runtime code that will be executed on the cluster.
+
+The filled arrows show how objects are transformed to other objects from one stage to the next stage during
+the translation process.
+
+<div style="text-align: center">
+  <img width="90%" src="{% link fig/table_connectors.svg %}" alt="Translation of table connectors" />
+</div>
+
+### Metadata
+
+Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing
+a `CREATE TABLE` statement results in updated metadata in the target catalog.
+
+For most catalog implementations, physical data in the external system is not modified for such an
+operation. Connector-specific dependencies must not be present in the classpath yet. The options declared
+in the `WITH` clause are neither validated nor otherwise interpreted.
+
+The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances
+of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary.
+
+### Planning
+
+When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved
+into a `DynamicTableSource` (for reading in a `SELECT` queries) and `DynamicTableSink` (for writing in
+a `INSERT INTO` statements).
+
+`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating
+the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most
+of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example),
+configure encoding/decoding formats (if required), and create a parameterized instance of the table
+connector.
+
+By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using
+Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The
+`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory
+identifier.
+
+Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink`
+can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing
+the actual data.
+
+The planner uses the source and sink instances to perform connector-specific bidirectional comunication
+until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g.
+`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and
+thus mutate the produced runtime implementation.
+
+### Runtime
+
+Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table
+connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`.
+
+Flink offers different core interfaces that are currently being unified. Therefore, the different runtime
+interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, `LookupRuntimeProvider`,
+and `SinkRuntimeProvider`.
+
+For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider`.
+
+{% top %}
+
+Extension Points
+----------------
+
+This section explains the available interfaces for extending Flink's table connectors.
+
+### Dynamic Table Factories
+
+Dynamic table factories configuring a dynamic table connector for an external storage system from catalog
+and session information.
+
+`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`.
+
+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`.
+
+By default, the factory is discovered using the value of the `connector` option as the factory identifier
+and Java's Service Provider Interface.
+
+In JAR files, references to new implementations can be added to the service file:
+
+`META-INF/services/org.apache.flink.table.factories.Factory`
+
+The framework will check for a single matching factory that is uniquely identified by factory identifier
+and requested base class (e.g. `DynamicTableSourceFactory`).
+
+The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a
+catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`.
+
+### Dynamic Table Source
+
+By definition, a dynamic table can change over time.
+
+When reading a dynamic table, the content can either be considered as:
+- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog
+  is exhausted. Represented in the `ScanTableSource` interface.
+- A continuously changing or very large external table whose content is usually never read entirely
+  but queried for individual values when necessary. Represented in the `LookupTableSource` interface.
+
+Both interfaces can be implemented at the same time. The planner decides about their usage depending
+on the specified query.
+
+#### Scan Table Source
+
+A `ScanTableSource` scans all rows from an external storage system during runtime.
+
+The scanned rows don't have to contain only insertions but can also contain updates and deletions. Thus,
+the table source can be used to read a (finite or infinite) changelog. The returned _changelog mode_ indicates
+the set of changes that the planner can expect during runtime.
+
+For regular batch scenarios, the source can emit a bounded stream of insert-only rows.
+
+For regular streaming scenarios, the source can emit an unbounded stream of insert-only rows.
+
+For change data capture (CDC) scenarios, the source can emit bounded or unbounded streams with insert,
+update, and delete rows.
+
+A table source can implement further abilitiy interfaces such as `SupportsProjectionPushDown` that might
+mutate an instance during planning. All abilities are listed in the `org.apache.flink.table.connector.source.abilities`
+package and in the documentation of `org.apache.flink.table.connector.source.ScanTableSource`.
+
+The runtime implementation of a `ScanTableSource` must produce internal data structures. Thus, records
+must be emitted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such
+that a source can still work on common data structures and perform a conversion at the end.
+
+#### Lookup Table Source
+
+A `LookupTableSource` looks up rows of an external storage system by one or more keys during runtime.
+
+Compared to `ScanTableSource`, the source must not read the entire table and can lazily fetch individual

Review comment:
       ```suggestion
   Compared to `ScanTableSource`, the source does not have to read the entire table and can lazily fetch individual
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 45594051f1293ba9d2309188224e055193ca4387 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920) 
   * 3265b3982f2240469fa9a102040bbb4a343ea932 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646656663


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 7776887183ee01b9b7f05ef4d3afeb892679faff (Fri Jun 19 14:08:39 UTC 2020)
   
   **Warnings:**
    * **1 pom.xml files were touched**: Check for build and licensing issues.
    * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket.
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3922",
       "triggerID" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 45594051f1293ba9d2309188224e055193ca4387 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920) 
   * 3265b3982f2240469fa9a102040bbb4a343ea932 UNKNOWN
   * bd428f54c9928ad024d0a89125a088ebccbf19d4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3922) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3922",
       "triggerID" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3265b3982f2240469fa9a102040bbb4a343ea932 UNKNOWN
   * bd428f54c9928ad024d0a89125a088ebccbf19d4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3922) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr closed pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
twalthr closed pull request #12724:
URL: https://github.com/apache/flink/pull/12724


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 45594051f1293ba9d2309188224e055193ca4387 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7776887183ee01b9b7f05ef4d3afeb892679faff UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-646670893


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3835",
       "triggerID" : "7776887183ee01b9b7f05ef4d3afeb892679faff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45594051f1293ba9d2309188224e055193ca4387",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920",
       "triggerID" : "45594051f1293ba9d2309188224e055193ca4387",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3265b3982f2240469fa9a102040bbb4a343ea932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bd428f54c9928ad024d0a89125a088ebccbf19d4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 45594051f1293ba9d2309188224e055193ca4387 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3920) 
   * 3265b3982f2240469fa9a102040bbb4a343ea932 UNKNOWN
   * bd428f54c9928ad024d0a89125a088ebccbf19d4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] aljoscha commented on pull request #12724: [FLINK-18006][FLINK-18380][docs][examples] Add documentation for new table sources and sinks

Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #12724:
URL: https://github.com/apache/flink/pull/12724#issuecomment-647598114


   Looks good to merge now!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org