You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2018/07/27 02:22:40 UTC
[GitHub] liurenjie1024 closed pull request #5688: [FLINK-6968][Table API &
SQL] Add Queryable table sink.
liurenjie1024 closed pull request #5688: [FLINK-6968][Table API & SQL] Add Queryable table sink.
URL: https://github.com/apache/flink/pull/5688
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index 8fc9fce6fb2..1fde2d9e4a1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -34,3 +34,4 @@ out/
*.iws
tools/flink
tools/flink-*
+tools/releasing/release
diff --git a/.travis.yml b/.travis.yml
index 4acdeb09df6..cad9c875f77 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -112,7 +112,7 @@ before_install:
- "export PATH=$M2_HOME/bin:$PATH"
- "export MAVEN_OPTS=\"-Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS\""
# just in case: clean up the .m2 home and remove invalid jar files
- - 'test ! -d $HOME/.m2/repository/ || find $HOME/.m2/repository/ -name "*.jar" -exec sh -c ''if ! zip -T {} >/dev/null ; then echo "deleting invalid file: {}"; rm {} ; fi'' \;'
+ - 'test ! -d $HOME/.m2/repository/ || find $HOME/.m2/repository/ -name "*.jar" -exec sh -c ''if ! zip -T {} >/dev/null ; then echo "deleting invalid file: {}"; rm -f {} ; fi'' \;'
# We run mvn and monitor its output. If there is no output for the specified number of seconds, we
# print the stack traces of all running Java processes.
diff --git a/LICENSE b/LICENSE
index f66290ed5ed..64850739a25 100644
--- a/LICENSE
+++ b/LICENSE
@@ -320,31 +320,3 @@ ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUT
DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.
-
------------------------------------------------------------------------
- BSD-style Licenses (BSD-2-clause)
------------------------------------------------------------------------
-
-The Apache Flink project contains or reuses code that is licensed under the BSD-2-clause license
-from the following projects:
- - jline-terminal (https://github.com/jline/jline3) Copyright (c) 2002-2016, various authors.
- - jline-reader (https://github.com/jline/jline3) Copyright (c) 2002-2016, various authors.
-
-Redistribution and use in source and binary forms, with or without modification, are permitted
-provided that the following conditions are met:
-
-1. Redistributions of source code must retain the above copyright notice, this list of conditions
- and the following disclaimer.
-
-2. Redistributions in binary form must reproduce the above copyright notice, this list of
- conditions and the following disclaimer in the documentation and/or other materials provided
- with the distribution.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
-IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
-FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
-CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
-IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
-OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.md b/README.md
index 095f652a354..7966ecd3d76 100644
--- a/README.md
+++ b/README.md
@@ -67,10 +67,10 @@ counts.writeAsCsv(outputPath)
Prerequisites for building Flink:
-* Unix-like environment (We use Linux, Mac OS X, Cygwin)
+* Unix-like environment (we use Linux, Mac OS X, Cygwin)
* git
* Maven (we recommend version 3.2.5)
-* Java 8
+* Java 8 (Java 9 and 10 are not yet supported)
```
git clone https://github.com/apache/flink.git
diff --git a/docs/_config.yml b/docs/_config.yml
index 605740878ca..983039fbf51 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -27,11 +27,12 @@
# we change the version for the complete docs when forking of a release branch
# etc.
# The full version string as referenced in Maven (e.g. 1.2.1)
-version: "1.6-SNAPSHOT"
+version: "1.7-SNAPSHOT"
# For stable releases, leave the bugfix version out (e.g. 1.2). For snapshot
# release this should be the same as the regular version
-version_title: "1.6-SNAPSHOT"
-version_javadocs: "1.6-SNAPSHOT"
+version_title: "1.7-SNAPSHOT"
+version_javadocs: "1.7-SNAPSHOT"
+version_scaladocs: "1.7-SNAPSHOT"
# This suffix is appended to the Scala-dependent Maven artifact names
scala_version_suffix: "_2.11"
@@ -53,6 +54,7 @@ is_stable: false
show_outdated_warning: false
previous_docs:
+ 1.5: http://ci.apache.org/projects/flink/flink-docs-release-1.5
1.4: http://ci.apache.org/projects/flink/flink-docs-release-1.4
1.3: http://ci.apache.org/projects/flink/flink-docs-release-1.3
1.2: http://ci.apache.org/projects/flink/flink-docs-release-1.2
diff --git a/docs/_includes/generated/akka_configuration.html b/docs/_includes/generated/akka_configuration.html
index afaba6f0fa5..352c656488e 100644
--- a/docs/_includes/generated/akka_configuration.html
+++ b/docs/_includes/generated/akka_configuration.html
@@ -80,17 +80,17 @@
<tr>
<td><h5>akka.watch.heartbeat.interval</h5></td>
<td style="word-wrap: break-word;">"10 s"</td>
- <td>Heartbeat interval for Akka’s DeathWatch mechanism to detect dead TaskManagers. If TaskManagers are wrongly marked dead because of lost or delayed heartbeat messages, then you should decrease this value or increase akka.watch.heartbeat.pause. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a>.</td>
+ <td>Heartbeat interval for Akka’s DeathWatch mechanism to detect dead TaskManagers. If TaskManagers are wrongly marked dead because of lost or delayed heartbeat messages, then you should decrease this value or increase akka.watch.heartbeat.pause. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a></td>
</tr>
<tr>
<td><h5>akka.watch.heartbeat.pause</h5></td>
<td style="word-wrap: break-word;">"60 s"</td>
- <td>Acceptable heartbeat pause for Akka’s DeathWatch mechanism. A low value does not allow an irregular heartbeat. If TaskManagers are wrongly marked dead because of lost or delayed heartbeat messages, then you should increase this value or decrease akka.watch.heartbeat.interval. Higher value increases the time to detect a dead TaskManager. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a>.</td>
+ <td>Acceptable heartbeat pause for Akka’s DeathWatch mechanism. A low value does not allow an irregular heartbeat. If TaskManagers are wrongly marked dead because of lost or delayed heartbeat messages, then you should increase this value or decrease akka.watch.heartbeat.interval. Higher value increases the time to detect a dead TaskManager. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a></td>
</tr>
<tr>
<td><h5>akka.watch.threshold</h5></td>
<td style="word-wrap: break-word;">12</td>
- <td>Threshold for the DeathWatch failure detector. A low value is prone to false positives whereas a high value increases the time to detect a dead TaskManager. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a>.</td>
+ <td>Threshold for the DeathWatch failure detector. A low value is prone to false positives whereas a high value increases the time to detect a dead TaskManager. A thorough description of Akka’s DeathWatch can be found <a href="http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector">here</a></td>
</tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/checkpointing_configuration.html b/docs/_includes/generated/checkpointing_configuration.html
index c64112e1f0c..8f5ce7b341d 100644
--- a/docs/_includes/generated/checkpointing_configuration.html
+++ b/docs/_includes/generated/checkpointing_configuration.html
@@ -28,14 +28,14 @@
<td>Option whether the state backend should create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state. Some state backends may not support incremental checkpoints and ignore this option.</td>
</tr>
<tr>
- <td><h5>state.backend.rocksdb.localdir</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>The local directory (on the TaskManager) where RocksDB puts its files.</td>
+ <td><h5>state.backend.local-recovery</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td></td>
</tr>
<tr>
<td><h5>state.checkpoints.dir</h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>The default directory used for checkpoints. Used by the state backends that write checkpoints to file systems (MemoryStateBackend, FsStateBackend, RocksDBStateBackend).</td>
+ <td>The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers).</td>
</tr>
<tr>
<td><h5>state.checkpoints.num-retained</h5></td>
@@ -47,5 +47,10 @@
<td style="word-wrap: break-word;">(none)</td>
<td>The default directory for savepoints. Used by the state backends that write savepoints to file systems (MemoryStateBackend, FsStateBackend, RocksDBStateBackend).</td>
</tr>
+ <tr>
+ <td><h5>taskmanager.state.local.root-dirs</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td></td>
+ </tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/common_section.html b/docs/_includes/generated/common_section.html
new file mode 100644
index 00000000000..ea881e0e347
--- /dev/null
+++ b/docs/_includes/generated/common_section.html
@@ -0,0 +1,66 @@
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 65%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>jobmanager.heap.size</h5></td>
+ <td style="word-wrap: break-word;">"1024m"</td>
+ <td>JVM heap size for the JobManager.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td style="word-wrap: break-word;">"1024m"</td>
+ <td>JVM heap size for the TaskManagers, which are the parallel workers of the system. On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.</td>
+ </tr>
+ <tr>
+ <td><h5>parallelism.default</h5></td>
+ <td style="word-wrap: break-word;">1</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.numberOfTaskSlots</h5></td>
+ <td style="word-wrap: break-word;">1</td>
+ <td>The number of parallel operator or user function instances that a single TaskManager can run. If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores).</td>
+ </tr>
+ <tr>
+ <td><h5>state.backend</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The state backend to be used to store and checkpoint state.</td>
+ </tr>
+ <tr>
+ <td><h5>state.checkpoints.dir</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers).</td>
+ </tr>
+ <tr>
+ <td><h5>state.savepoints.dir</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The default directory for savepoints. Used by the state backends that write savepoints to file systems (MemoryStateBackend, FsStateBackend, RocksDBStateBackend).</td>
+ </tr>
+ <tr>
+ <td><h5>high-availability</h5></td>
+ <td style="word-wrap: break-word;">"NONE"</td>
+ <td>Defines high-availability mode used for the cluster execution. To enable high-availability, set this mode to "ZOOKEEPER".</td>
+ </tr>
+ <tr>
+ <td><h5>high-availability.storageDir</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>File system path (URI) where Flink persists metadata in high-availability setups.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.enabled</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Turns on SSL for internal network communication. Optionally, specific components may override this through their own settings (rpc, data transport, REST, etc).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.enabled</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Turns on SSL for external communication via the REST endpoints.</td>
+ </tr>
+ </tbody>
+</table>
diff --git a/docs/_includes/generated/core_configuration.html b/docs/_includes/generated/core_configuration.html
index 520ce05a541..98cca9125a0 100644
--- a/docs/_includes/generated/core_configuration.html
+++ b/docs/_includes/generated/core_configuration.html
@@ -8,9 +8,14 @@
</thead>
<tbody>
<tr>
- <td><h5>classloader.parent-first-patterns</h5></td>
- <td style="word-wrap: break-word;">"java.;scala.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.slf4j;org.apache.log4j;org.apache.logging.log4j;ch.qos.logback"</td>
- <td>A (semicolon-separated) list of patterns that specifies which classes should always be resolved through the parent ClassLoader first. A pattern is a simple prefix that is checked against the fully qualified class name.</td>
+ <td><h5>classloader.parent-first-patterns.additional</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>A (semicolon-separated) list of patterns that specifies which classes should always be resolved through the parent ClassLoader first. A pattern is a simple prefix that is checked against the fully qualified class name. These patterns are appended to "classloader.parent-first-patterns.default".</td>
+ </tr>
+ <tr>
+ <td><h5>classloader.parent-first-patterns.default</h5></td>
+ <td style="word-wrap: break-word;">"java.;<wbr>scala.;<wbr>org.apache.flink.;<wbr>com.esotericsoftware.kryo;<wbr>org.apache.hadoop.;<wbr>javax.annotation.;<wbr>org.slf4j;<wbr>org.apache.log4j;<wbr>org.apache.logging;<wbr>org.apache.commons.logging;<wbr>ch.qos.logback"</td>
+ <td>A (semicolon-separated) list of patterns that specifies which classes should always be resolved through the parent ClassLoader first. A pattern is a simple prefix that is checked against the fully qualified class name. This setting should generally not be modified. To add another pattern we recommend to use "classloader.parent-first-patterns.additional" instead.</td>
</tr>
<tr>
<td><h5>classloader.resolve-order</h5></td>
@@ -19,13 +24,13 @@
</tr>
<tr>
<td><h5>io.tmp.dirs</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style="word-wrap: break-word;">'LOCAL_DIRS' on Yarn. '_FLINK_TMP_DIR' on Mesos. System.getProperty("java.io.tmpdir") in standalone.</td>
<td></td>
</tr>
<tr>
<td><h5>mode</h5></td>
- <td style="word-wrap: break-word;">"flip6"</td>
- <td>Switch to select the execution mode. Possible values are 'flip6' and 'old'.</td>
+ <td style="word-wrap: break-word;">"new"</td>
+ <td>Switch to select the execution mode. Possible values are 'new' and 'legacy'.</td>
</tr>
<tr>
<td><h5>parallelism.default</h5></td>
diff --git a/docs/_includes/generated/job_manager_configuration.html b/docs/_includes/generated/job_manager_configuration.html
index 251731a8e05..0353874c9b6 100644
--- a/docs/_includes/generated/job_manager_configuration.html
+++ b/docs/_includes/generated/job_manager_configuration.html
@@ -23,9 +23,9 @@
<td>The maximum number of prior execution attempts kept in history.</td>
</tr>
<tr>
- <td><h5>jobmanager.heap.mb</h5></td>
- <td style="word-wrap: break-word;">1024</td>
- <td>JVM heap size (in megabytes) for the JobManager.</td>
+ <td><h5>jobmanager.heap.size</h5></td>
+ <td style="word-wrap: break-word;">"1024m"</td>
+ <td>JVM heap size for the JobManager.</td>
</tr>
<tr>
<td><h5>jobmanager.resourcemanager.reconnect-interval</h5></td>
@@ -54,7 +54,7 @@
</tr>
<tr>
<td><h5>slot.idle.timeout</h5></td>
- <td style="word-wrap: break-word;">10000</td>
+ <td style="word-wrap: break-word;">50000</td>
<td>The timeout in milliseconds for a idle slot in Slot Pool.</td>
</tr>
<tr>
diff --git a/docs/_includes/generated/mesos_configuration.html b/docs/_includes/generated/mesos_configuration.html
index 16a23886cee..cd0ae2432e3 100644
--- a/docs/_includes/generated/mesos_configuration.html
+++ b/docs/_includes/generated/mesos_configuration.html
@@ -62,5 +62,10 @@
<td style="word-wrap: break-word;">(none)</td>
<td>Mesos framework user</td>
</tr>
+ <tr>
+ <td><h5>mesos.resourcemanager.tasks.port-assignments</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Comma-separated list of configuration keys which represent a configurable port.All port keys will dynamically get a port assigned through Mesos.</td>
+ </tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/mesos_task_manager_configuration.html b/docs/_includes/generated/mesos_task_manager_configuration.html
index a6ee78b60fd..0af844dbceb 100644
--- a/docs/_includes/generated/mesos_task_manager_configuration.html
+++ b/docs/_includes/generated/mesos_task_manager_configuration.html
@@ -17,6 +17,11 @@
<td style="word-wrap: break-word;">(none)</td>
<td></td>
</tr>
+ <tr>
+ <td><h5>mesos.resourcemanager.tasks.container.docker.force-pull-image</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Instruct the docker containerizer to forcefully pull the image rather than reuse a cached version.</td>
+ </tr>
<tr>
<td><h5>mesos.resourcemanager.tasks.container.docker.parameters</h5></td>
<td style="word-wrap: break-word;">(none)</td>
@@ -62,6 +67,11 @@
<td style="word-wrap: break-word;">"$FLINK_HOME/bin/mesos-taskmanager.sh"</td>
<td></td>
</tr>
+ <tr>
+ <td><h5>mesos.resourcemanager.tasks.uris</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>A comma separated list of URIs of custom artifacts to be downloaded into the sandbox of Mesos workers.</td>
+ </tr>
<tr>
<td><h5>taskmanager.numberOfTaskSlots</h5></td>
<td style="word-wrap: break-word;">1</td>
diff --git a/docs/_includes/generated/metric_configuration.html b/docs/_includes/generated/metric_configuration.html
index 868a9bd66e9..aef8fbb4f60 100644
--- a/docs/_includes/generated/metric_configuration.html
+++ b/docs/_includes/generated/metric_configuration.html
@@ -13,19 +13,19 @@
<td>Defines the number of measured latencies to maintain at each operator.</td>
</tr>
<tr>
- <td><h5>metrics.reporter.<name>.<parameter></h5></td>
+ <td><h5>metrics.reporter.<name>.<parameter></h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>Configures the parameter <parameter> for the reporter named <name>.</td>
+ <td>Configures the parameter <parameter> for the reporter named <name>.</td>
</tr>
<tr>
- <td><h5>metrics.reporter.<name>.class</h5></td>
+ <td><h5>metrics.reporter.<name>.class</h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>The reporter class to use for the reporter named <name>.</td>
+ <td>The reporter class to use for the reporter named <name>.</td>
</tr>
<tr>
- <td><h5>metrics.reporter.<name>.interval</h5></td>
+ <td><h5>metrics.reporter.<name>.interval</h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>The reporter interval to use for the reporter named <name>.</td>
+ <td>The reporter interval to use for the reporter named <name>.</td>
</tr>
<tr>
<td><h5>metrics.reporters</h5></td>
@@ -39,32 +39,32 @@
</tr>
<tr>
<td><h5>metrics.scope.jm</h5></td>
- <td style="word-wrap: break-word;">"<host>.jobmanager"</td>
+ <td style="word-wrap: break-word;">"<host>.jobmanager"</td>
<td>Defines the scope format string that is applied to all metrics scoped to a JobManager.</td>
</tr>
<tr>
<td><h5>metrics.scope.jm.job</h5></td>
- <td style="word-wrap: break-word;">"<host>.jobmanager.<job_name>"</td>
+ <td style="word-wrap: break-word;">"<host>.jobmanager.<job_name>"</td>
<td>Defines the scope format string that is applied to all metrics scoped to a job on a JobManager.</td>
</tr>
<tr>
<td><h5>metrics.scope.operator</h5></td>
- <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>"</td>
+ <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>"</td>
<td>Defines the scope format string that is applied to all metrics scoped to an operator.</td>
</tr>
<tr>
<td><h5>metrics.scope.task</h5></td>
- <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>.<task_name>.<subtask_index>"</td>
+ <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>.<task_name>.<subtask_index>"</td>
<td>Defines the scope format string that is applied to all metrics scoped to a task.</td>
</tr>
<tr>
<td><h5>metrics.scope.tm</h5></td>
- <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>"</td>
+ <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>"</td>
<td>Defines the scope format string that is applied to all metrics scoped to a TaskManager.</td>
</tr>
<tr>
<td><h5>metrics.scope.tm.job</h5></td>
- <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>"</td>
+ <td style="word-wrap: break-word;">"<host>.taskmanager.<tm_id>.<job_name>"</td>
<td>Defines the scope format string that is applied to all metrics scoped to a job on a TaskManager.</td>
</tr>
</tbody>
diff --git a/docs/_includes/generated/netty_configuration.html b/docs/_includes/generated/netty_configuration.html
index e97fda45d10..47c48c0aa38 100644
--- a/docs/_includes/generated/netty_configuration.html
+++ b/docs/_includes/generated/netty_configuration.html
@@ -42,20 +42,5 @@
<td style="word-wrap: break-word;">"nio"</td>
<td>The Netty transport type, either "nio" or "epoll"</td>
</tr>
- <tr>
- <td><h5>taskmanager.network.credit-based-flow-control.enabled</h5></td>
- <td style="word-wrap: break-word;">true</td>
- <td>Boolean flag to enable/disable network credit-based flow control</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.memory.buffers-per-channel</h5></td>
- <td style="word-wrap: break-word;">2</td>
- <td>Number of network buffers to use for each outgoing/incoming channel (subpartition/input channel). In credit-based flow control mode, this indicates how many credits are exclusive in each input channel. It should be configured at least 2 for good performance. 1 buffer is for receiving in-flight data in the subpartition and 1 buffer is for parallel serialization</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.memory.floating-buffers-per-gate</h5></td>
- <td style="word-wrap: break-word;">8</td>
- <td>Number of extra network buffers to use for each outgoing/incoming gate (result partition/input gate). In credit-based flow control mode, this indicates how many floating credits are shared among all the input channels. The floating buffers are distributed based on backlog (real-time output buffers in the subpartition) feedback, and can help relieve back-pressure caused by unbalanced data distribution among the subpartitions. This value should be increased in case of higher round trip times between nodes and/or larger number of machines in the cluster</td>
- </tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/prometheus_push_gateway_reporter_configuration.html b/docs/_includes/generated/prometheus_push_gateway_reporter_configuration.html
new file mode 100644
index 00000000000..2d14b50777b
--- /dev/null
+++ b/docs/_includes/generated/prometheus_push_gateway_reporter_configuration.html
@@ -0,0 +1,36 @@
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 65%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>deleteOnShutdown</h5></td>
+ <td style="word-wrap: break-word;">true</td>
+ <td>Specifies whether to delete metrics from the PushGateway on shutdown.</td>
+ </tr>
+ <tr>
+ <td><h5>host</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The PushGateway server host.</td>
+ </tr>
+ <tr>
+ <td><h5>jobName</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The job name under which metrics will be pushed</td>
+ </tr>
+ <tr>
+ <td><h5>port</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>The PushGateway server port.</td>
+ </tr>
+ <tr>
+ <td><h5>randomJobNameSuffix</h5></td>
+ <td style="word-wrap: break-word;">true</td>
+ <td>Specifies whether a random suffix should be appended to the job name.</td>
+ </tr>
+ </tbody>
+</table>
diff --git a/docs/_includes/generated/rest_configuration.html b/docs/_includes/generated/rest_configuration.html
index 20fe96138a5..1de41654eb7 100644
--- a/docs/_includes/generated/rest_configuration.html
+++ b/docs/_includes/generated/rest_configuration.html
@@ -9,14 +9,24 @@
<tbody>
<tr>
<td><h5>rest.address</h5></td>
- <td style="word-wrap: break-word;">"localhost"</td>
- <td>The address that the server binds itself to / the client connects to.</td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The address that should be used by clients to connect to the server.</td>
</tr>
<tr>
<td><h5>rest.await-leader-timeout</h5></td>
<td style="word-wrap: break-word;">30000</td>
<td>The time in ms that the client waits for the leader address, e.g., Dispatcher or WebMonitorEndpoint</td>
</tr>
+ <tr>
+ <td><h5>rest.bind-address</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The address that the server binds itself.</td>
+ </tr>
+ <tr>
+ <td><h5>rest.client.max-content-length</h5></td>
+ <td style="word-wrap: break-word;">104857600</td>
+ <td>The maximum content length in bytes that the client will handle.</td>
+ </tr>
<tr>
<td><h5>rest.connection-timeout</h5></td>
<td style="word-wrap: break-word;">15000</td>
@@ -24,7 +34,7 @@
</tr>
<tr>
<td><h5>rest.port</h5></td>
- <td style="word-wrap: break-word;">9065</td>
+ <td style="word-wrap: break-word;">8081</td>
<td>The port that the server listens on / the client connects to.</td>
</tr>
<tr>
@@ -37,5 +47,10 @@
<td style="word-wrap: break-word;">20</td>
<td>The number of retries the client will attempt if a retryable operations fails.</td>
</tr>
+ <tr>
+ <td><h5>rest.server.max-content-length</h5></td>
+ <td style="word-wrap: break-word;">104857600</td>
+ <td>The maximum content length in bytes that the server will handle.</td>
+ </tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/rest_dispatcher.html b/docs/_includes/generated/rest_dispatcher.html
index 8ac36e1faae..ab0f4e0a0a4 100644
--- a/docs/_includes/generated/rest_dispatcher.html
+++ b/docs/_includes/generated/rest_dispatcher.html
@@ -1,19 +1,19 @@
<table class="table table-bordered">
<tbody>
<tr>
- <td class="text-left" colspan="2"><strong>/blobserver/port</strong></td>
+ <td class="text-left" colspan="2"><strong>/cluster</strong></td>
</tr>
<tr>
- <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
+ <td class="text-left" style="width: 20%">Verb: <code>DELETE</code></td>
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Shuts down the cluster</td>
</tr>
<tr>
<td colspan="2">
- <button data-toggle="collapse" data-target="#607508253">Request</button>
- <div id="607508253" class="collapse">
+ <button data-toggle="collapse" data-target="#-1674237828">Request</button>
+ <div id="-1674237828" class="collapse">
<pre>
<code>
{} </code>
@@ -23,19 +23,11 @@
</tr>
<tr>
<td colspan="2">
- <button data-toggle="collapse" data-target="#1913718109">Response</button>
- <div id="1913718109" class="collapse">
+ <button data-toggle="collapse" data-target="#-1489255004">Response</button>
+ <div id="-1489255004" class="collapse">
<pre>
<code>
-{
- "type" : "object",
- "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:BlobServerPortResponseBody",
- "properties" : {
- "port" : {
- "type" : "integer"
- }
- }
-} </code>
+{} </code>
</pre>
</div>
</td>
@@ -52,7 +44,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the configuration of the WebUI.</td>
</tr>
<tr>
<td colspan="2">
@@ -98,6 +90,326 @@
</tr>
</tbody>
</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jars</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Returns a list of all jars previously uploaded via '/jars/upload'.</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-525177469">Request</button>
+ <div id="-525177469" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-1669843955">Response</button>
+ <div id="-1669843955" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarListInfo",
+ "properties" : {
+ "address" : {
+ "type" : "string"
+ },
+ "files" : {
+ "type" : "array",
+ "items" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarListInfo:JarFileInfo",
+ "properties" : {
+ "id" : {
+ "type" : "string"
+ },
+ "name" : {
+ "type" : "string"
+ },
+ "uploaded" : {
+ "type" : "integer"
+ },
+ "entry" : {
+ "type" : "array",
+ "items" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarListInfo:JarEntryInfo",
+ "properties" : {
+ "name" : {
+ "type" : "string"
+ },
+ "description" : {
+ "type" : "string"
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jars/upload</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>POST</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Uploads a jar to the cluster. The jar must be sent as multi-part data. Make sure that the "Content-Type" header is set to "application/x-java-archive", as some http libraries do not add the header by default.
+Using 'curl' you can upload a jar via 'curl -X POST -H "Expect:" -F "jarfile=#path/to/flink-job.jar" http://hostname:port/jars/upload'.</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-1290030289">Request</button>
+ <div id="-1290030289" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#1132420209">Response</button>
+ <div id="1132420209" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarUploadResponseBody",
+ "properties" : {
+ "filename" : {
+ "type" : "string"
+ },
+ "status" : {
+ "type" : "string",
+ "enum" : [ "success" ]
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jars/:jarid</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>DELETE</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Deletes a jar previously uploaded via '/jars/upload'.</td>
+ </tr>
+ <tr>
+ <td colspan="2">Path parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>jarid</code> - description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-974639435">Request</button>
+ <div id="-974639435" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-789656611">Response</button>
+ <div id="-789656611" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jars/:jarid/plan</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Returns the dataflow plan of a job contained in a jar previously uploaded via '/jars/upload'.</td>
+ </tr>
+ <tr>
+ <td colspan="2">Path parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>jarid</code> - description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">Query parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>entry-class</code> (optional): description</li>
+<li><code>parallelism</code> (optional): description</li>
+<li><code>program-args</code> (optional): description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-181694384">Request</button>
+ <div id="-181694384" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#1311035100">Response</button>
+ <div id="1311035100" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "any"
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jars/:jarid/run</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>POST</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Submits a job by running a jar previously uploaded via '/jars/upload'.</td>
+ </tr>
+ <tr>
+ <td colspan="2">Path parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>jarid</code> - description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">Query parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>program-args</code> (optional): description</li>
+<li><code>entry-class</code> (optional): description</li>
+<li><code>parallelism</code> (optional): description</li>
+<li><code>allowNonRestoredState</code> (optional): description</li>
+<li><code>savepointPath</code> (optional): description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#315035146">Request</button>
+ <div id="315035146" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarRunRequestBody",
+ "properties" : {
+ "entryClass" : {
+ "type" : "string"
+ },
+ "programArgs" : {
+ "type" : "string"
+ },
+ "parallelism" : {
+ "type" : "integer"
+ },
+ "allowNonRestoredState" : {
+ "type" : "boolean"
+ },
+ "savepointPath" : {
+ "type" : "string"
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-914083094">Response</button>
+ <div id="-914083094" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarRunResponseBody",
+ "properties" : {
+ "jobid" : {
+ "type" : "any"
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
<table class="table table-bordered">
<tbody>
<tr>
@@ -108,7 +420,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the cluster configuration.</td>
</tr>
<tr>
<td colspan="2">
@@ -158,7 +470,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to job manager metrics.</td>
</tr>
<tr>
<td colspan="2">Query parameters</td>
@@ -206,7 +518,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns an overview over all jobs and their current state.</td>
</tr>
<tr>
<td colspan="2">
@@ -263,7 +575,7 @@
<td class="text-left">Response code: <code>202 Accepted</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Submits a job. This call is primarily intended to be used by the Flink client. This call expects amultipart/form-data request that consists of file uploads for the serialized JobGraph, jars anddistributed cache artifacts and an attribute named "request"for the JSON payload.</td>
</tr>
<tr>
<td colspan="2">
@@ -275,10 +587,28 @@
"type" : "object",
"id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:JobSubmitRequestBody",
"properties" : {
- "serializedJobGraph" : {
+ "jobGraphFileName" : {
+ "type" : "string"
+ },
+ "jobJarFileNames" : {
"type" : "array",
"items" : {
- "type" : "integer"
+ "type" : "string"
+ }
+ },
+ "jobArtifactFileNames" : {
+ "type" : "array",
+ "items" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:JobSubmitRequestBody:DistributedCacheFile",
+ "properties" : {
+ "entryName" : {
+ "type" : "string"
+ },
+ "fileName" : {
+ "type" : "string"
+ }
+ }
}
}
}
@@ -318,7 +648,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to aggregated job metrics.</td>
</tr>
<tr>
<td colspan="2">Query parameters</td>
@@ -368,7 +698,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns an overview over all jobs.</td>
</tr>
<tr>
<td colspan="2">
@@ -415,7 +745,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns details of a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -573,7 +903,7 @@
<td class="text-left">Response code: <code>202 Accepted</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Terminates a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -629,7 +959,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the accumulators for all tasks of a job, aggregated across the respective subtasks.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -720,7 +1050,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns checkpointing statistics for a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1023,7 +1353,85 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the checkpointing configuration.</td>
+ </tr>
+ <tr>
+ <td colspan="2">Path parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>jobid</code> - description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#1905242249">Request</button>
+ <div id="1905242249" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#808099872">Response</button>
+ <div id="808099872" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointConfigInfo",
+ "properties" : {
+ "mode" : {
+ "type" : "any"
+ },
+ "interval" : {
+ "type" : "integer"
+ },
+ "timeout" : {
+ "type" : "integer"
+ },
+ "min_pause" : {
+ "type" : "integer"
+ },
+ "max_concurrent" : {
+ "type" : "integer"
+ },
+ "externalization" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointConfigInfo:ExternalizedCheckpointInfo",
+ "properties" : {
+ "enabled" : {
+ "type" : "boolean"
+ },
+ "delete_on_cancellation" : {
+ "type" : "boolean"
+ }
+ }
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jobs/:jobid/checkpoints/details/:checkpointid</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Returns details for a checkpoint.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1032,13 +1440,14 @@
<td colspan="2">
<ul>
<li><code>jobid</code> - description</li>
+<li><code>checkpointid</code> - description</li>
</ul>
</td>
</tr>
<tr>
<td colspan="2">
- <button data-toggle="collapse" data-target="#1905242249">Request</button>
- <div id="1905242249" class="collapse">
+ <button data-toggle="collapse" data-target="#2082812513">Request</button>
+ <div id="2082812513" class="collapse">
<pre>
<code>
{} </code>
@@ -1048,38 +1457,76 @@
</tr>
<tr>
<td colspan="2">
- <button data-toggle="collapse" data-target="#808099872">Response</button>
- <div id="808099872" class="collapse">
+ <button data-toggle="collapse" data-target="#59893771">Response</button>
+ <div id="59893771" class="collapse">
<pre>
<code>
{
"type" : "object",
- "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointConfigInfo",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointStatistics",
"properties" : {
- "mode" : {
- "type" : "any"
+ "id" : {
+ "type" : "integer"
},
- "interval" : {
+ "status" : {
+ "type" : "string",
+ "enum" : [ "IN_PROGRESS", "COMPLETED", "FAILED" ]
+ },
+ "is_savepoint" : {
+ "type" : "boolean"
+ },
+ "trigger_timestamp" : {
"type" : "integer"
},
- "timeout" : {
+ "latest_ack_timestamp" : {
"type" : "integer"
},
- "min_pause" : {
+ "state_size" : {
"type" : "integer"
},
- "max_concurrent" : {
+ "end_to_end_duration" : {
"type" : "integer"
},
- "externalization" : {
+ "alignment_buffered" : {
+ "type" : "integer"
+ },
+ "num_subtasks" : {
+ "type" : "integer"
+ },
+ "num_acknowledged_subtasks" : {
+ "type" : "integer"
+ },
+ "tasks" : {
"type" : "object",
- "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointConfigInfo:ExternalizedCheckpointInfo",
- "properties" : {
- "enabled" : {
- "type" : "boolean"
- },
- "delete_on_cancellation" : {
- "type" : "boolean"
+ "additionalProperties" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:TaskCheckpointStatistics",
+ "properties" : {
+ "id" : {
+ "type" : "integer"
+ },
+ "status" : {
+ "type" : "string",
+ "enum" : [ "IN_PROGRESS", "COMPLETED", "FAILED" ]
+ },
+ "latest_ack_timestamp" : {
+ "type" : "integer"
+ },
+ "state_size" : {
+ "type" : "integer"
+ },
+ "end_to_end_duration" : {
+ "type" : "integer"
+ },
+ "alignment_buffered" : {
+ "type" : "integer"
+ },
+ "num_subtasks" : {
+ "type" : "integer"
+ },
+ "num_acknowledged_subtasks" : {
+ "type" : "integer"
+ }
}
}
}
@@ -1101,7 +1548,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns checkpoint statistics for a task and its subtasks.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1237,123 +1684,6 @@
</tr>
</tbody>
</table>
-<table class="table table-bordered">
- <tbody>
- <tr>
- <td class="text-left" colspan="2"><strong>/jobs/:jobid/checkpoints/:checkpointid</strong></td>
- </tr>
- <tr>
- <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
- <td class="text-left">Response code: <code>200 OK</code></td>
- </tr>
- <tr>
- <td colspan="2">description</td>
- </tr>
- <tr>
- <td colspan="2">Path parameters</td>
- </tr>
- <tr>
- <td colspan="2">
- <ul>
-<li><code>jobid</code> - description</li>
-<li><code>checkpointid</code> - description</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td colspan="2">
- <button data-toggle="collapse" data-target="#-968188818">Request</button>
- <div id="-968188818" class="collapse">
- <pre>
- <code>
-{} </code>
- </pre>
- </div>
- </td>
- </tr>
- <tr>
- <td colspan="2">
- <button data-toggle="collapse" data-target="#1303859736">Response</button>
- <div id="1303859736" class="collapse">
- <pre>
- <code>
-{
- "type" : "object",
- "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:CheckpointStatistics",
- "properties" : {
- "id" : {
- "type" : "integer"
- },
- "status" : {
- "type" : "string",
- "enum" : [ "IN_PROGRESS", "COMPLETED", "FAILED" ]
- },
- "is_savepoint" : {
- "type" : "boolean"
- },
- "trigger_timestamp" : {
- "type" : "integer"
- },
- "latest_ack_timestamp" : {
- "type" : "integer"
- },
- "state_size" : {
- "type" : "integer"
- },
- "end_to_end_duration" : {
- "type" : "integer"
- },
- "alignment_buffered" : {
- "type" : "integer"
- },
- "num_subtasks" : {
- "type" : "integer"
- },
- "num_acknowledged_subtasks" : {
- "type" : "integer"
- },
- "tasks" : {
- "type" : "object",
- "additionalProperties" : {
- "type" : "object",
- "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:checkpoints:TaskCheckpointStatistics",
- "properties" : {
- "id" : {
- "type" : "integer"
- },
- "status" : {
- "type" : "string",
- "enum" : [ "IN_PROGRESS", "COMPLETED", "FAILED" ]
- },
- "latest_ack_timestamp" : {
- "type" : "integer"
- },
- "state_size" : {
- "type" : "integer"
- },
- "end_to_end_duration" : {
- "type" : "integer"
- },
- "alignment_buffered" : {
- "type" : "integer"
- },
- "num_subtasks" : {
- "type" : "integer"
- },
- "num_acknowledged_subtasks" : {
- "type" : "integer"
- }
- }
- }
- }
- }
-} </code>
- </pre>
- </div>
- </td>
- </tr>
- </tbody>
-</table>
<table class="table table-bordered">
<tbody>
<tr>
@@ -1364,7 +1694,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the configuration of a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1412,7 +1742,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the non-recoverable exceptions that have been observed by the job. The truncated flag defines whether more exceptions occurred, but are not listed, because the response would otherwise get too big.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1493,7 +1823,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the result of a job execution. Gives access to the execution time of the job and to all accumulators created by this job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1559,7 +1889,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to job metrics.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1617,7 +1947,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the dataflow plan of a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1665,7 +1995,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Triggers the rescaling of a job.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1729,7 +2059,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the status of a rescaling operation.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1795,7 +2125,7 @@
<td class="text-left">Response code: <code>202 Accepted</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Triggers a savepoint, and optionally cancels the job afterwards.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1860,7 +2190,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the status of a savepoint operation.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -1926,7 +2256,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns details for a task, with a summary for each of its subtasks.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2051,7 +2381,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns user-defined accumulators of a task, aggregated across all subtasks.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2124,7 +2454,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns back-pressure information for a job, and may initiate back-pressure sampling if necessary.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2206,7 +2536,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to task metrics.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2255,6 +2585,100 @@
</tr>
</tbody>
</table>
+<table class="table table-bordered">
+ <tbody>
+ <tr>
+ <td class="text-left" colspan="2"><strong>/jobs/:jobid/vertices/:vertexid/subtasks/accumulators</strong></td>
+ </tr>
+ <tr>
+ <td class="text-left" style="width: 20%">Verb: <code>GET</code></td>
+ <td class="text-left">Response code: <code>200 OK</code></td>
+ </tr>
+ <tr>
+ <td colspan="2">Returns all user-defined accumulators for all subtasks of a task.</td>
+ </tr>
+ <tr>
+ <td colspan="2">Path parameters</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <ul>
+<li><code>jobid</code> - description</li>
+<li><code>vertexid</code> - description</li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#-886388859">Request</button>
+ <div id="-886388859" class="collapse">
+ <pre>
+ <code>
+{} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <button data-toggle="collapse" data-target="#112317594">Response</button>
+ <div id="112317594" class="collapse">
+ <pre>
+ <code>
+{
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtasksAllAccumulatorsInfo",
+ "properties" : {
+ "id" : {
+ "type" : "any"
+ },
+ "parallelism" : {
+ "type" : "integer"
+ },
+ "subtasks" : {
+ "type" : "array",
+ "items" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtasksAllAccumulatorsInfo:SubtaskAccumulatorsInfo",
+ "properties" : {
+ "subtask" : {
+ "type" : "integer"
+ },
+ "attempt" : {
+ "type" : "integer"
+ },
+ "host" : {
+ "type" : "string"
+ },
+ "user-accumulators" : {
+ "type" : "array",
+ "items" : {
+ "type" : "object",
+ "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:UserAccumulator",
+ "properties" : {
+ "name" : {
+ "type" : "string"
+ },
+ "type" : {
+ "type" : "string"
+ },
+ "value" : {
+ "type" : "string"
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+} </code>
+ </pre>
+ </div>
+ </td>
+ </tr>
+ </tbody>
+</table>
<table class="table table-bordered">
<tbody>
<tr>
@@ -2265,7 +2689,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to aggregated subtask metrics.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2326,7 +2750,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns details of the current or latest execution attempt of a subtask.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2431,7 +2855,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns details of an execution attempt of a subtask. Multiple execution attempts happen in case of failure/recovery.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2537,7 +2961,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the accumulators of an execution attempt of a subtask. Multiple execution attempts happen in case of failure/recovery.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2618,7 +3042,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to subtask metrics.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2678,7 +3102,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns time-related information for all subtasks of a task.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2763,7 +3187,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns task information aggregated by task manager.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -2885,7 +3309,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns an overview over the Flink cluster.</td>
</tr>
<tr>
<td colspan="2">
@@ -2953,7 +3377,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Triggers the desposal of a savepoint.</td>
</tr>
<tr>
<td colspan="2">
@@ -3005,7 +3429,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns the status of a savepoint disposal operation.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -3070,7 +3494,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns an overview over all task managers.</td>
</tr>
<tr>
<td colspan="2">
@@ -3156,7 +3580,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to aggregated task manager metrics.</td>
</tr>
<tr>
<td colspan="2">Query parameters</td>
@@ -3206,7 +3630,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Returns details for a task manager.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
@@ -3359,7 +3783,7 @@
<td class="text-left">Response code: <code>200 OK</code></td>
</tr>
<tr>
- <td colspan="2">description</td>
+ <td colspan="2">Provides access to task manager metrics.</td>
</tr>
<tr>
<td colspan="2">Path parameters</td>
diff --git a/docs/_includes/generated/rocks_db_configuration.html b/docs/_includes/generated/rocks_db_configuration.html
new file mode 100644
index 00000000000..57b95114976
--- /dev/null
+++ b/docs/_includes/generated/rocks_db_configuration.html
@@ -0,0 +1,21 @@
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 65%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>state.backend.rocksdb.localdir</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The local directory (on the TaskManager) where RocksDB puts its files.</td>
+ </tr>
+ <tr>
+ <td><h5>state.backend.rocksdb.timer-service.impl</h5></td>
+ <td style="word-wrap: break-word;">"HEAP"</td>
+ <td>This determines the timer service implementation. Options are either HEAP (heap-based, default) or ROCKSDB for an implementation based on RocksDB.</td>
+ </tr>
+ </tbody>
+</table>
diff --git a/docs/_includes/generated/security_configuration.html b/docs/_includes/generated/security_configuration.html
index cd682ecaf0f..19d0287df99 100644
--- a/docs/_includes/generated/security_configuration.html
+++ b/docs/_includes/generated/security_configuration.html
@@ -10,12 +10,37 @@
<tr>
<td><h5>security.ssl.algorithms</h5></td>
<td style="word-wrap: break-word;">"TLS_RSA_WITH_AES_128_CBC_SHA"</td>
- <td>The comma separated list of standard SSL algorithms to be supported. Read more <a href="http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites">here</a>.</td>
+ <td>The comma separated list of standard SSL algorithms to be supported. Read more <a href="http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites">here</a></td>
</tr>
<tr>
- <td><h5>security.ssl.enabled</h5></td>
+ <td><h5>security.ssl.internal.enabled</h5></td>
<td style="word-wrap: break-word;">false</td>
- <td>Turns on SSL for internal network communication. This can be optionally overridden by flags defined in different transport modules.</td>
+ <td>Turns on SSL for internal network communication. Optionally, specific components may override this through their own settings (rpc, data transport, REST, etc).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.key-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The secret to decrypt the key in the keystore for Flink's internal endpoints (rpc, data transport, blob server).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.keystore</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The Java keystore file with SSL Key and Certificate, to be used Flink's internal endpoints (rpc, data transport, blob server).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.keystore-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The secret to decrypt the keystore file for Flink's for Flink's internal endpoints (rpc, data transport, blob server).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.truststore</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The truststore file containing the public CA certificates to verify the peer for Flink's internal endpoints (rpc, data transport, blob server).</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.internal.truststore-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The password to decrypt the truststore for Flink's internal endpoints (rpc, data transport, blob server).</td>
</tr>
<tr>
<td><h5>security.ssl.key-password</h5></td>
@@ -37,6 +62,36 @@
<td style="word-wrap: break-word;">"TLSv1.2"</td>
<td>The SSL protocol version to be supported for the ssl transport. Note that it doesn’t support comma separated list.</td>
</tr>
+ <tr>
+ <td><h5>security.ssl.rest.enabled</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Turns on SSL for external communication via the REST endpoints.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.key-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The secret to decrypt the key in the keystore for Flink's external REST endpoints.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.keystore</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The Java keystore file with SSL Key and Certificate, to be used Flink's external REST endpoints.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.keystore-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The secret to decrypt the keystore file for Flink's for Flink's external REST endpoints.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.truststore</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The truststore file containing the public CA certificates to verify the peer for Flink's external REST endpoints.</td>
+ </tr>
+ <tr>
+ <td><h5>security.ssl.rest.truststore-password</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>The password to decrypt the truststore for Flink's external REST endpoints.</td>
+ </tr>
<tr>
<td><h5>security.ssl.truststore</h5></td>
<td style="word-wrap: break-word;">(none)</td>
diff --git a/docs/_includes/generated/task_manager_configuration.html b/docs/_includes/generated/task_manager_configuration.html
index bd422445dfb..c18c5d4c36d 100644
--- a/docs/_includes/generated/task_manager_configuration.html
+++ b/docs/_includes/generated/task_manager_configuration.html
@@ -35,53 +35,38 @@
<tr>
<td><h5>taskmanager.data.ssl.enabled</h5></td>
<td style="word-wrap: break-word;">true</td>
- <td>Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true</td>
+ <td>Enable SSL support for the taskmanager data transport. This is applicable only when the global flag for internal SSL (security.ssl.internal.enabled) is set to true</td>
</tr>
<tr>
- <td><h5>taskmanager.debug.memory.logIntervalMs</h5></td>
- <td style="word-wrap: break-word;">5000</td>
- <td>The interval (in ms) for the log thread to log the current memory usage.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.debug.memory.startLogThread</h5></td>
+ <td><h5>taskmanager.debug.memory.log</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Flag indicating whether to start a thread, which repeatedly logs the memory usage of the JVM.</td>
</tr>
+ <tr>
+ <td><h5>taskmanager.debug.memory.log-interval</h5></td>
+ <td style="word-wrap: break-word;">5000</td>
+ <td>The interval (in ms) for the log thread to log the current memory usage.</td>
+ </tr>
<tr>
<td><h5>taskmanager.exit-on-fatal-akka-error</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Whether the quarantine monitor for task managers shall be started. The quarantine monitor shuts down the actor system if it detects that it has quarantined another actor system or if it has been quarantined by another actor system.</td>
</tr>
<tr>
- <td><h5>taskmanager.heap.mb</h5></td>
- <td style="word-wrap: break-word;">1024</td>
- <td>JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.</td>
+ <td><h5>taskmanager.heap.size</h5></td>
+ <td style="word-wrap: break-word;">"1024m"</td>
+ <td>JVM heap size for the TaskManagers, which are the parallel workers of the system. On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.</td>
</tr>
<tr>
<td><h5>taskmanager.host</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file.</td>
</tr>
- <tr>
- <td><h5>taskmanager.initial-registration-pause</h5></td>
- <td style="word-wrap: break-word;">"500 ms"</td>
- <td>The initial registration pause between two consecutive registration attempts. The pause is doubled for each new registration attempt until it reaches the maximum registration pause.</td>
- </tr>
<tr>
<td><h5>taskmanager.jvm-exit-on-oom</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Whether to kill the TaskManager when the task thread throws an OutOfMemoryError.</td>
</tr>
- <tr>
- <td><h5>taskmanager.max-registration-pause</h5></td>
- <td style="word-wrap: break-word;">"30 s"</td>
- <td>The maximum registration pause between two consecutive registration attempts. The max registration pause requires a time unit specifier (ms/s/min/h/d).</td>
- </tr>
- <tr>
- <td><h5>taskmanager.maxRegistrationDuration</h5></td>
- <td style="word-wrap: break-word;">"Inf"</td>
- <td>Defines the maximum time it can take for the TaskManager registration. If the duration is exceeded without a successful registration, then the TaskManager terminates.</td>
- </tr>
<tr>
<td><h5>taskmanager.memory.fraction</h5></td>
<td style="word-wrap: break-word;">0.7</td>
@@ -99,13 +84,13 @@
</tr>
<tr>
<td><h5>taskmanager.memory.segment-size</h5></td>
- <td style="word-wrap: break-word;">32768</td>
- <td>Size of memory buffers used by the network stack and the memory manager (in bytes).</td>
+ <td style="word-wrap: break-word;">"32768"</td>
+ <td>Size of memory buffers used by the network stack and the memory manager.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.size</h5></td>
- <td style="word-wrap: break-word;">-1</td>
- <td>Amount of memory to be allocated by the task manager's memory manager (in megabytes). If not set, a relative fraction will be allocated.</td>
+ <td style="word-wrap: break-word;">"0"</td>
+ <td>Amount of memory to be allocated by the task manager's memory manager. If not set, a relative fraction will be allocated.</td>
</tr>
<tr>
<td><h5>taskmanager.network.detailed-metrics</h5></td>
@@ -115,12 +100,12 @@
<tr>
<td><h5>taskmanager.network.memory.buffers-per-channel</h5></td>
<td style="word-wrap: break-word;">2</td>
- <td>Number of network buffers to use for each outgoing/incoming channel (subpartition/input channel).</td>
+ <td>Maximum number of network buffers to use for each outgoing/incoming channel (subpartition/input channel).In credit-based flow control mode, this indicates how many credits are exclusive in each input channel. It should be configured at least 2 for good performance. 1 buffer is for receiving in-flight data in the subpartition and 1 buffer is for parallel serialization.</td>
</tr>
<tr>
<td><h5>taskmanager.network.memory.floating-buffers-per-gate</h5></td>
<td style="word-wrap: break-word;">8</td>
- <td>Number of extra network buffers to use for each outgoing/incoming gate (result partition/input gate).</td>
+ <td>Number of extra network buffers to use for each outgoing/incoming gate (result partition/input gate). In credit-based flow control mode, this indicates how many floating credits are shared among all the input channels. The floating buffers are distributed based on backlog (real-time output buffers in the subpartition) feedback, and can help relieve back-pressure caused by unbalanced data distribution among the subpartitions. This value should be increased in case of higher round trip times between nodes and/or larger number of machines in the cluster.</td>
</tr>
<tr>
<td><h5>taskmanager.network.memory.fraction</h5></td>
@@ -129,13 +114,13 @@
</tr>
<tr>
<td><h5>taskmanager.network.memory.max</h5></td>
- <td style="word-wrap: break-word;">1073741824</td>
- <td>Maximum memory size for network buffers (in bytes).</td>
+ <td style="word-wrap: break-word;">"1073741824"</td>
+ <td>Maximum memory size for network buffers.</td>
</tr>
<tr>
<td><h5>taskmanager.network.memory.min</h5></td>
- <td style="word-wrap: break-word;">67108864</td>
- <td>Minimum memory size for network buffers (in bytes).</td>
+ <td style="word-wrap: break-word;">"67108864"</td>
+ <td>Minimum memory size for network buffers.</td>
</tr>
<tr>
<td><h5>taskmanager.network.request-backoff.initial</h5></td>
@@ -153,9 +138,24 @@
<td>The number of parallel operator or user function instances that a single TaskManager can run. If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores).</td>
</tr>
<tr>
- <td><h5>taskmanager.refused-registration-pause</h5></td>
+ <td><h5>taskmanager.registration.initial-backoff</h5></td>
+ <td style="word-wrap: break-word;">"500 ms"</td>
+ <td>The initial registration backoff between two consecutive registration attempts. The backoff is doubled for each new registration attempt until it reaches the maximum registration backoff.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.registration.max-backoff</h5></td>
+ <td style="word-wrap: break-word;">"30 s"</td>
+ <td>The maximum registration backoff between two consecutive registration attempts. The max registration backoff requires a time unit specifier (ms/s/min/h/d).</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.registration.refused-backoff</h5></td>
<td style="word-wrap: break-word;">"10 s"</td>
- <td>The pause after a registration has been refused by the job manager before retrying to connect.</td>
+ <td>The backoff after a registration has been refused by the job manager before retrying to connect.</td>
+ </tr>
+ <tr>
+ <td><h5>taskmanager.registration.timeout</h5></td>
+ <td style="word-wrap: break-word;">"5 min"</td>
+ <td>Defines the timeout for the TaskManager registration. If the duration is exceeded without a successful registration, then the TaskManager terminates.</td>
</tr>
<tr>
<td><h5>taskmanager.rpc.port</h5></td>
diff --git a/docs/_includes/generated/web_configuration.html b/docs/_includes/generated/web_configuration.html
index 23bf5ab63cb..dbc353dbd56 100644
--- a/docs/_includes/generated/web_configuration.html
+++ b/docs/_includes/generated/web_configuration.html
@@ -52,11 +52,6 @@
<td style="word-wrap: break-word;">(none)</td>
<td></td>
</tr>
- <tr>
- <td><h5>web.port</h5></td>
- <td style="word-wrap: break-word;">8081</td>
- <td></td>
- </tr>
<tr>
<td><h5>web.refresh-interval</h5></td>
<td style="word-wrap: break-word;">3000</td>
@@ -79,7 +74,7 @@
</tr>
<tr>
<td><h5>web.tmpdir</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style="word-wrap: break-word;">System.getProperty("java.io.tmpdir")</td>
<td></td>
</tr>
<tr>
diff --git a/docs/_includes/sidenav.html b/docs/_includes/sidenav.html
index 7afc5f1f398..5e50c350d6b 100644
--- a/docs/_includes/sidenav.html
+++ b/docs/_includes/sidenav.html
@@ -127,6 +127,7 @@
{% endfor %}
<li class="divider"></li>
<li><a href="https://ci.apache.org/projects/flink/flink-docs-release-{{site.version_javadocs}}/api/java"><i class="fa fa-external-link title" aria-hidden="true"></i> Javadocs</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-{{site.version_scaladocs}}/api/scala/index.html#org.apache.flink.api.scala.package"><i class="fa fa-external-link title" aria-hidden="true"></i> Scaladocs</a></li>
<li><a href="http://flink.apache.org"><i class="fa fa-external-link title" aria-hidden="true"></i> Project Page</a></li>
</ul>
diff --git a/docs/dev/batch/connectors.md b/docs/dev/batch/connectors.md
index 23917448f39..76771378f49 100644
--- a/docs/dev/batch/connectors.md
+++ b/docs/dev/batch/connectors.md
@@ -64,12 +64,12 @@ See [Deployment & Operations - Deployment - AWS - S3: Simple Storage Service]({{
For Alluxio support add the following entry into the `core-site.xml` file:
-~~~xml
+{% highlight xml %}
<property>
<name>fs.alluxio.impl</name>
<value>alluxio.hadoop.FileSystem</value>
</property>
-~~~
+{% endhighlight %}
## Connecting to other systems using Input/OutputFormat wrappers for Hadoop
@@ -89,28 +89,28 @@ This section shows some examples for connecting Flink to other systems.
Flink has extensive build-in support for [Apache Avro](http://avro.apache.org/). This allows to easily read from Avro files with Flink.
Also, the serialization framework of Flink is able to handle classes generated from Avro schemas. Be sure to include the Flink Avro dependency to the pom.xml of your project.
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
<version>{{site.version }}</version>
</dependency>
-~~~
+{% endhighlight %}
In order to read data from an Avro file, you have to specify an `AvroInputFormat`.
**Example**:
-~~~java
+{% highlight java %}
AvroInputFormat<User> users = new AvroInputFormat<User>(in, User.class);
DataSet<User> usersDS = env.createInput(users);
-~~~
+{% endhighlight %}
Note that `User` is a POJO generated by Avro. Flink also allows to perform string-based key selection of these POJOs. For example:
-~~~java
+{% highlight java %}
usersDS.groupBy("name")
-~~~
+{% endhighlight %}
Note that using the `GenericData.Record` type is possible with Flink, but not recommended. Since the record contains the full schema, its very data intensive and thus probably slow to use.
@@ -129,21 +129,21 @@ This example is using the `HadoopInputFormat` wrapper to use an existing Hadoop
1. Download and compile the `azure-tables-hadoop` project. The input format developed by the project is not yet available in Maven Central, therefore, we have to build the project ourselves.
Execute the following commands:
- ~~~bash
+ {% highlight bash %}
git clone https://github.com/mooso/azure-tables-hadoop.git
cd azure-tables-hadoop
mvn clean install
- ~~~
+ {% endhighlight %}
2. Setup a new Flink project using the quickstarts:
- ~~~bash
+ {% highlight bash %}
curl https://flink.apache.org/q/quickstart.sh | bash
- ~~~
+ {% endhighlight %}
3. Add the following dependencies (in the `<dependencies>` section) to your `pom.xml` file:
- ~~~xml
+ {% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-compatibility{{ site.scala_version_suffix }}</artifactId>
@@ -154,7 +154,7 @@ Execute the following commands:
<artifactId>microsoft-hadoop-azure</artifactId>
<version>0.0.4</version>
</dependency>
- ~~~
+ {% endhighlight %}
`flink-hadoop-compatibility` is a Flink package that provides the Hadoop input format wrappers.
`microsoft-hadoop-azure` is adding the project we've build before to our project.
@@ -164,7 +164,7 @@ Browse to the code of the `Job.java` file. Its an empty skeleton for a Flink job
Paste the following code into it:
-~~~java
+{% highlight java %}
import java.util.Map;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.DataSet;
@@ -217,7 +217,7 @@ public class AzureTableExample {
env.execute("Azure Example");
}
}
-~~~
+{% endhighlight %}
The example shows how to access an Azure table and turn data into Flink's `DataSet` (more specifically, the type of the set is `DataSet<Tuple2<Text, WritableEntity>>`). With the `DataSet`, you can apply all known transformations to the DataSet.
diff --git a/docs/dev/batch/dataset_transformations.md b/docs/dev/batch/dataset_transformations.md
index d63ee88ccd1..96c04c9d548 100644
--- a/docs/dev/batch/dataset_transformations.md
+++ b/docs/dev/batch/dataset_transformations.md
@@ -42,7 +42,7 @@ The following code transforms a DataSet of Integer pairs into a DataSet of Integ
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// MapFunction that adds two integer values
public class IntAdder implements MapFunction<Tuple2<Integer, Integer>, Integer> {
@Override
@@ -54,22 +54,22 @@ public class IntAdder implements MapFunction<Tuple2<Integer, Integer>, Integer>
// [...]
DataSet<Tuple2<Integer, Integer>> intPairs = // [...]
DataSet<Integer> intSums = intPairs.map(new IntAdder());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val intPairs: DataSet[(Int, Int)] = // [...]
val intSums = intPairs.map { pair => pair._1 + pair._2 }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
intSums = intPairs.map(lambda x: sum(x))
-~~~
+{% endhighlight %}
</div>
</div>
@@ -84,7 +84,7 @@ The following code transforms a DataSet of text lines into a DataSet of words:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// FlatMapFunction that tokenizes a String by whitespace characters and emits all String tokens.
public class Tokenizer implements FlatMapFunction<String, String> {
@Override
@@ -98,22 +98,22 @@ public class Tokenizer implements FlatMapFunction<String, String> {
// [...]
DataSet<String> textLines = // [...]
DataSet<String> words = textLines.flatMap(new Tokenizer());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val textLines: DataSet[String] = // [...]
val words = textLines.flatMap { _.split(" ") }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
words = lines.flat_map(lambda x,c: [line.split() for line in x])
-~~~
+{% endhighlight %}
</div>
</div>
@@ -129,7 +129,7 @@ The following code transforms a DataSet of text lines into a DataSet of counts p
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class PartitionCounter implements MapPartitionFunction<String, Long> {
public void mapPartition(Iterable<String> values, Collector<Long> out) {
@@ -144,24 +144,24 @@ public class PartitionCounter implements MapPartitionFunction<String, Long> {
// [...]
DataSet<String> textLines = // [...]
DataSet<Long> counts = textLines.mapPartition(new PartitionCounter());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val textLines: DataSet[String] = // [...]
// Some is required because the return value must be a Collection.
// There is an implicit conversion from Option to a Collection.
val counts = texLines.mapPartition { in => Some(in.size) }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
counts = lines.map_partition(lambda x,c: [sum(1 for _ in x)])
-~~~
+{% endhighlight %}
</div>
</div>
@@ -175,7 +175,7 @@ The following code removes all Integers smaller than zero from a DataSet:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// FilterFunction that filters out all Integers smaller than zero.
public class NaturalNumberFilter implements FilterFunction<Integer> {
@Override
@@ -187,22 +187,22 @@ public class NaturalNumberFilter implements FilterFunction<Integer> {
// [...]
DataSet<Integer> intNumbers = // [...]
DataSet<Integer> naturalNumbers = intNumbers.filter(new NaturalNumberFilter());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val intNumbers: DataSet[Int] = // [...]
val naturalNumbers = intNumbers.filter { _ > 0 }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
naturalNumbers = intNumbers.filter(lambda x: x > 0)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -222,40 +222,40 @@ The following code shows different ways to apply a Project transformation on a D
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, Double, String>> in = // [...]
// converts Tuple3<Integer, Double, String> into Tuple2<String, Integer>
DataSet<Tuple2<String, Integer>> out = in.project(2,0);
-~~~
+{% endhighlight %}
#### Projection with Type Hint
Note that the Java compiler cannot infer the return type of `project` operator. This can cause a problem if you call another operator on a result of `project` operator such as:
-~~~java
+{% highlight java %}
DataSet<Tuple5<String,String,String,String,String>> ds = ....
DataSet<Tuple1<String>> ds2 = ds.project(0).distinct(0);
-~~~
+{% endhighlight %}
This problem can be overcome by hinting the return type of `project` operator like this:
-~~~java
+{% highlight java %}
DataSet<Tuple1<String>> ds2 = ds.<Tuple1<String>>project(0).distinct(0);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
Not supported.
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
out = in.project(2,0);
-~~~
+{% endhighlight %}
</div>
</div>
@@ -294,7 +294,7 @@ with a reduce function.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// some ordinary POJO
public class WC {
public String word;
@@ -317,12 +317,12 @@ DataSet<WC> wordCounts = words
.groupBy("word")
// apply ReduceFunction on grouped DataSet
.reduce(new WordCounter());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// some ordinary POJO
class WC(val word: String, val count: Int) {
def this() {
@@ -335,14 +335,14 @@ val words: DataSet[WC] = // [...]
val wordCounts = words.groupBy("word").reduce {
(w1, w2) => new WC(w1.word, w1.count + w2.count)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -356,7 +356,7 @@ with a reduce function.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// some ordinary POJO
public class WC {
public String word;
@@ -386,12 +386,12 @@ public class SelectWord implements KeySelector<WC, String> {
return w.word;
}
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// some ordinary POJO
class WC(val word: String, val count: Int) {
def this() {
@@ -404,12 +404,12 @@ val words: DataSet[WC] = // [...]
val wordCounts = words.groupBy { _.word } reduce {
(w1, w2) => new WC(w1.word, w1.count + w2.count)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class WordCounter(ReduceFunction):
def reduce(self, in1, in2):
return (in1[0], in1[1] + in2[1])
@@ -418,7 +418,7 @@ words = // [...]
wordCounts = words \
.group_by(lambda x: x[0]) \
.reduce(WordCounter())
-~~~
+{% endhighlight %}
</div>
</div>
@@ -430,30 +430,30 @@ The following code shows how to use field position keys and apply a reduce funct
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<String, Integer, Double>> tuples = // [...]
DataSet<Tuple3<String, Integer, Double>> reducedTuples = tuples
// group DataSet on first and second field of Tuple
.groupBy(0, 1)
// apply ReduceFunction on grouped DataSet
.reduce(new MyTupleReducer());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val tuples = DataSet[(String, Int, Double)] = // [...]
// group on the first and second Tuple field
val reducedTuples = tuples.groupBy(0, 1).reduce { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
reducedTuples = tuples.group_by(0, 1).reduce( ... )
-~~~
+{% endhighlight %}
</div>
</div>
@@ -465,25 +465,25 @@ When using Case Classes you can also specify the grouping key using the names of
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
Not supported.
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
case class MyClass(val a: String, b: Int, c: Double)
val tuples = DataSet[MyClass] = // [...]
// group on the first and second field
val reducedTuples = tuples.groupBy("a", "b").reduce { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -502,7 +502,7 @@ The following code shows how duplicate strings can be removed from a DataSet gro
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class DistinctReduce
implements GroupReduceFunction<Tuple2<Integer, String>, Tuple2<Integer, String>> {
@@ -530,23 +530,23 @@ DataSet<Tuple2<Integer, String>> input = // [...]
DataSet<Tuple2<Integer, String>> output = input
.groupBy(0) // group DataSet by the first tuple field
.reduceGroup(new DistinctReduce()); // apply GroupReduceFunction
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String)] = // [...]
val output = input.groupBy(0).reduceGroup {
(in, out: Collector[(Int, String)]) =>
in.toSet foreach (out.collect)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class DistinctReduce(GroupReduceFunction):
def reduce(self, iterator, collector):
dic = dict()
@@ -556,7 +556,7 @@ val output = input.groupBy(0).reduceGroup {
collector.collect(key)
output = data.group_by(0).reduce_group(DistinctReduce())
-~~~
+{% endhighlight %}
</div>
</div>
@@ -578,7 +578,7 @@ The following code shows another example how to remove duplicate Strings in a Da
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// GroupReduceFunction that removes consecutive identical elements
public class DistinctReduce
implements GroupReduceFunction<Tuple2<Integer, String>, Tuple2<Integer, String>> {
@@ -607,12 +607,12 @@ DataSet<Double> output = input
.groupBy(0) // group DataSet by first field
.sortGroup(1, Order.ASCENDING) // sort groups on second tuple field
.reduceGroup(new DistinctReduce());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String)] = // [...]
val output = input.groupBy(0).sortGroup(1, Order.ASCENDING).reduceGroup {
(in, out: Collector[(Int, String)]) =>
@@ -624,12 +624,12 @@ val output = input.groupBy(0).sortGroup(1, Order.ASCENDING).reduceGroup {
}
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class DistinctReduce(GroupReduceFunction):
def reduce(self, iterator, collector):
dic = dict()
@@ -639,7 +639,7 @@ val output = input.groupBy(0).sortGroup(1, Order.ASCENDING).reduceGroup {
collector.collect(key)
output = data.group_by(0).sort_group(1, Order.ASCENDING).reduce_group(DistinctReduce())
-~~~
+{% endhighlight %}
</div>
@@ -660,7 +660,7 @@ of the `GroupReduceFunction` as shown in the following example:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// Combinable GroupReduceFunction that computes a sum.
public class MyCombinableGroupReducer implements
GroupReduceFunction<Tuple2<String, Integer>, String>,
@@ -695,12 +695,12 @@ public class MyCombinableGroupReducer implements
out.collect(new Tuple2<>(key, sum));
}
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// Combinable GroupReduceFunction that computes two sums.
class MyCombinableGroupReducer
@@ -727,12 +727,12 @@ class MyCombinableGroupReducer
out.collect(r)
}
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class GroupReduce(GroupReduceFunction):
def reduce(self, iterator, collector):
key, int_sum = iterator.next()
@@ -747,7 +747,7 @@ class MyCombinableGroupReducer
collector.collect((key, int_sum))
data.reduce_group(GroupReduce(), combinable=True)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -778,7 +778,7 @@ an alternative WordCount implementation.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<String> input = [..] // The words received as input
DataSet<Tuple2<String, Integer>> combinedWords = input
@@ -814,12 +814,12 @@ DataSet<Tuple2<String, Integer>> output = combinedWords
out.collect(new Tuple2(key, count));
}
});
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[String] = [..] // The words received as input
val combinedWords: DataSet[(String, Int)] = input
@@ -850,14 +850,14 @@ val output: DataSet[(String, Int)] = combinedWords
out.collect((key, sum))
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -883,31 +883,31 @@ The following code shows how to apply an Aggregation transformation on a DataSet
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, String, Double>> input = // [...]
DataSet<Tuple3<Integer, String, Double>> output = input
.groupBy(1) // group DataSet on second field
.aggregate(SUM, 0) // compute sum of the first field
.and(MIN, 2); // compute minimum of the third field
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String, Double)] = // [...]
val output = input.groupBy(1).aggregate(SUM, 0).and(MIN, 2)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
from flink.functions.Aggregation import Sum, Min
input = # [...]
output = input.group_by(1).aggregate(Sum, 0).and_agg(Min, 2)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -926,29 +926,29 @@ The following code shows how to select the tuple with the minimum values for the
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, String, Double>> input = // [...]
DataSet<Tuple3<Integer, String, Double>> output = input
.groupBy(1) // group DataSet on second field
.minBy(0, 2); // select tuple with minimum values for first and third field.
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String, Double)] = // [...]
val output: DataSet[(Int, String, Double)] = input
.groupBy(1) // group DataSet on second field
.minBy(0, 2) // select tuple with minimum values for first and third field.
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -963,7 +963,7 @@ The following code shows how to sum all elements of an Integer DataSet:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// ReduceFunction that sums Integers
public class IntSummer implements ReduceFunction<Integer> {
@Override
@@ -975,23 +975,23 @@ public class IntSummer implements ReduceFunction<Integer> {
// [...]
DataSet<Integer> intNumbers = // [...]
DataSet<Integer> sum = intNumbers.reduce(new IntSummer());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val intNumbers = env.fromElements(1,2,3)
val sum = intNumbers.reduce (_ + _)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
intNumbers = env.from_elements(1,2,3)
sum = intNumbers.reduce(lambda x,y: x + y)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1008,26 +1008,26 @@ The following example shows how to apply a GroupReduce transformation on a full
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Integer> input = // [...]
// apply a (preferably combinable) GroupReduceFunction to a DataSet
DataSet<Double> output = input.reduceGroup(new MyGroupReducer());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[Int] = // [...]
val output = input.reduceGroup(new MyGroupReducer())
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
output = data.reduce_group(MyGroupReducer())
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1059,31 +1059,31 @@ The following code shows how to apply an Aggregation transformation on a full Da
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<Integer, Double>> input = // [...]
DataSet<Tuple2<Integer, Double>> output = input
.aggregate(SUM, 0) // compute sum of the first field
.and(MIN, 1); // compute minimum of the second field
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String, Double)] = // [...]
val output = input.aggregate(SUM, 0).and(MIN, 2)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
from flink.functions.Aggregation import Sum, Min
input = # [...]
output = input.aggregate(Sum, 0).and_agg(Min, 2)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1099,27 +1099,27 @@ The following code shows how to select the tuple with the maximum values for the
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, String, Double>> input = // [...]
DataSet<Tuple3<Integer, String, Double>> output = input
.maxBy(0, 2); // select tuple with maximum values for first and third field.
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String, Double)] = // [...]
val output: DataSet[(Int, String, Double)] = input
.maxBy(0, 2) // select tuple with maximum values for first and third field.
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1132,27 +1132,27 @@ The following code removes all duplicate elements from the DataSet:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<Integer, Double>> input = // [...]
DataSet<Tuple2<Integer, Double>> output = input.distinct();
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, String, Double)] = // [...]
val output = input.distinct()
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1168,27 +1168,27 @@ It is also possible to change how the distinction of the elements in the DataSet
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<Integer, Double, String>> input = // [...]
DataSet<Tuple2<Integer, Double, String>> output = input.distinct(0,2);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[(Int, Double, String)] = // [...]
val output = input.distinct(0,2)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1198,7 +1198,7 @@ Not supported.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
private static class AbsSelector implements KeySelector<Integer, Integer> {
private static final long serialVersionUID = 1L;
@Override
@@ -1209,23 +1209,23 @@ private static final long serialVersionUID = 1L;
DataSet<Integer> input = // [...]
DataSet<Integer> output = input.distinct(new AbsSelector());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input: DataSet[Int] = // [...]
val output = input.distinct {x => Math.abs(x)}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1235,7 +1235,7 @@ Not supported.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// some ordinary POJO
public class CustomType {
public String aName;
@@ -1246,26 +1246,26 @@ public class CustomType {
DataSet<CustomType> input = // [...]
DataSet<CustomType> output = input.distinct("aName", "aNumber");
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// some ordinary POJO
case class CustomType(aName : String, aNumber : Int) { }
val input: DataSet[CustomType] = // [...]
val output = input.distinct("aName", "aNumber")
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1275,28 +1275,28 @@ It is also possible to indicate to use all the fields by the wildcard character:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<CustomType> input = // [...]
DataSet<CustomType> output = input.distinct("*");
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// some ordinary POJO
val input: DataSet[CustomType] = // [...]
val output = input.distinct("_")
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1321,7 +1321,7 @@ The following code shows a default Join transformation using field position keys
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public static class User { public String name; public int zip; }
public static class Store { public Manager mgr; public int zip; }
DataSet<User> input1 = // [...]
@@ -1331,23 +1331,23 @@ DataSet<Tuple2<User, Store>>
result = input1.join(input2)
.where("zip") // key of the first input (users)
.equalTo("zip"); // key of the second input (stores)
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input1: DataSet[(Int, String)] = // [...]
val input2: DataSet[(Double, Int)] = // [...]
val result = input1.join(input2).where(0).equalTo(1)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
result = input1.join(input2).where(0).equal_to(1)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1362,7 +1362,7 @@ The following code performs a join of DataSet with custom java objects and a Tup
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// some POJO
public class Rating {
public String name;
@@ -1395,12 +1395,12 @@ DataSet<Tuple2<String, Double>>
// applying the JoinFunction on joining pairs
.with(new PointWeighter());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
case class Rating(name: String, category: String, points: Int)
val ratings: DataSet[Ratings] = // [...]
@@ -1409,12 +1409,12 @@ val weights: DataSet[(String, Double)] = // [...]
val weightedRatings = ratings.join(weights).where("category").equalTo(0) {
(rating, weight) => (rating.name, rating.points * weight._2)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class PointWeighter(JoinFunction):
def join(self, rating, weight):
return (rating[0], rating[1] * weight[1])
@@ -1423,7 +1423,7 @@ val weightedRatings = ratings.join(weights).where("category").equalTo(0) {
weightedRatings =
ratings.join(weights).where(0).equal_to(0). \
with(new PointWeighter());
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1437,7 +1437,7 @@ return (collect), zero, one, or more elements.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class PointWeighter
implements FlatJoinFunction<Rating, Tuple2<String, Double>, Tuple2<String, Double>> {
@Override
@@ -1452,12 +1452,12 @@ public class PointWeighter
DataSet<Tuple2<String, Double>>
weightedRatings =
ratings.join(weights) // [...]
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
case class Rating(name: String, category: String, points: Int)
val ratings: DataSet[Ratings] = // [...]
@@ -1468,7 +1468,7 @@ val weightedRatings = ratings.join(weights).where("category").equalTo(0) {
if (weight._2 > 0.1) out.collect(rating.name, rating.points * weight._2)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
@@ -1483,10 +1483,10 @@ A Join transformation can construct result tuples using a projection as shown he
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, Byte, String>> input1 = // [...]
DataSet<Tuple2<Integer, Double>> input2 = // [...]
-DataSet<Tuple4<Integer, String, Double, Byte>
+DataSet<Tuple4<Integer, String, Double, Byte>>
result =
input1.join(input2)
// key definition on first DataSet using a field position key
@@ -1495,7 +1495,7 @@ DataSet<Tuple4<Integer, String, Double, Byte>
.equalTo(0)
// select and reorder fields of matching tuples
.projectFirst(0,2).projectSecond(1).projectFirst(1);
-~~~
+{% endhighlight %}
`projectFirst(int...)` and `projectSecond(int...)` select the fields of the first and second joined input that should be assembled into an output Tuple. The order of indexes defines the order of fields in the output tuple.
The join projection works also for non-Tuple DataSets. In this case, `projectFirst()` or `projectSecond()` must be called without arguments to add a joined element to the output Tuple.
@@ -1503,17 +1503,17 @@ The join projection works also for non-Tuple DataSets. In this case, `projectFir
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
Not supported.
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
result = input1.join(input2).where(0).equal_to(0) \
.project_first(0,2).project_second(1).project_first(1);
-~~~
+{% endhighlight %}
`project_first(int...)` and `project_second(int...)` select the fields of the first and second joined input that should be assembled into an output Tuple. The order of indexes defines the order of fields in the output tuple.
The join projection works also for non-Tuple DataSets. In this case, `project_first()` or `project_second()` must be called without arguments to add a joined element to the output Tuple.
@@ -1528,7 +1528,7 @@ In order to guide the optimizer to pick the right execution strategy, you can hi
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<Integer, String>> input1 = // [...]
DataSet<Tuple2<Integer, String>> input2 = // [...]
@@ -1545,12 +1545,12 @@ DataSet<Tuple2<Tuple2<Integer, String>, Tuple2<Integer, String>>>
input1.joinWithHuge(input2)
.where(0)
.equalTo(0);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input1: DataSet[(Int, String)] = // [...]
val input2: DataSet[(Int, String)] = // [...]
@@ -1560,12 +1560,12 @@ val result1 = input1.joinWithTiny(input2).where(0).equalTo(0)
// hint that the second DataSet is very large
val result1 = input1.joinWithHuge(input2).where(0).equalTo(0)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
#hint that the second DataSet is very small
result1 = input1.join_with_tiny(input2).where(0).equal_to(0)
@@ -1573,7 +1573,7 @@ val result1 = input1.joinWithHuge(input2).where(0).equalTo(0)
#hint that the second DataSet is very large
result1 = input1.join_with_huge(input2).where(0).equal_to(0)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1587,33 +1587,33 @@ to manually pick a strategy, in case you want to enforce a specific way of execu
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<SomeType> input1 = // [...]
DataSet<AnotherType> input2 = // [...]
DataSet<Tuple2<SomeType, AnotherType> result =
input1.join(input2, JoinHint.BROADCAST_HASH_FIRST)
.where("id").equalTo("key");
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input1: DataSet[SomeType] = // [...]
val input2: DataSet[AnotherType] = // [...]
// hint that the second DataSet is very small
val result1 = input1.join(input2, JoinHint.BROADCAST_HASH_FIRST).where("id").equalTo("key")
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1668,7 +1668,7 @@ The following code performs a left outer join of DataSet with custom java object
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// some POJO
public class Rating {
public String name;
@@ -1702,12 +1702,12 @@ DataSet<Tuple2<String, Integer>>
// applying the JoinFunction on joining pairs
.with(new PointAssigner());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
case class Rating(name: String, category: String, points: Int)
val movies: DataSet[(String, String)] = // [...]
@@ -1716,14 +1716,14 @@ val ratings: DataSet[Ratings] = // [...]
val moviesWithPoints = movies.leftOuterJoin(ratings).where(0).equalTo("name") {
(movie, rating) => (movie._1, if (rating == null) -1 else rating.points)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1737,7 +1737,7 @@ return (collect), zero, one, or more elements.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class PointAssigner
implements FlatJoinFunction<Tuple2<String, String>, Rating, Tuple2<String, Integer>> {
@Override
@@ -1755,21 +1755,21 @@ public class PointAssigner
DataSet<Tuple2<String, Integer>>
moviesWithPoints =
movies.leftOuterJoin(ratings) // [...]
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
Not supported.
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1783,7 +1783,7 @@ to manually pick a strategy, in case you want to enforce a specific way of execu
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<SomeType> input1 = // [...]
DataSet<AnotherType> input2 = // [...]
@@ -1794,12 +1794,12 @@ DataSet<Tuple2<SomeType, AnotherType> result1 =
DataSet<Tuple2<SomeType, AnotherType> result2 =
input1.rightOuterJoin(input2, JoinHint.BROADCAST_HASH_FIRST)
.where("id").equalTo("key");
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input1: DataSet[SomeType] = // [...]
val input2: DataSet[AnotherType] = // [...]
@@ -1808,14 +1808,14 @@ val result1 = input1.leftOuterJoin(input2, JoinHint.REPARTITION_SORT_MERGE).wher
val result2 = input1.rightOuterJoin(input2, JoinHint.BROADCAST_HASH_FIRST).where("id").equalTo("key")
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1878,7 +1878,7 @@ The following code shows how to apply a Cross transformation on two DataSets usi
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class Coord {
public int id;
public int x;
@@ -1904,13 +1904,13 @@ DataSet<Tuple3<Integer, Integer, Double>>
coords1.cross(coords2)
// apply CrossFunction
.with(new EuclideanDistComputer());
-~~~
+{% endhighlight %}
#### Cross with Projection
A Cross transformation can also construct result tuples using a projection as shown here:
-~~~java
+{% highlight java %}
DataSet<Tuple3<Integer, Byte, String>> input1 = // [...]
DataSet<Tuple2<Integer, Double>> input2 = // [...]
DataSet<Tuple4<Integer, Byte, Integer, Double>
@@ -1918,14 +1918,14 @@ DataSet<Tuple4<Integer, Byte, Integer, Double>
input1.cross(input2)
// select and reorder fields of matching tuples
.projectSecond(0).projectFirst(1,0).projectSecond(1);
-~~~
+{% endhighlight %}
The field selection in a Cross projection works the same way as in the projection of Join results.
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
case class Coord(id: Int, x: Int, y: Int)
val coords1: DataSet[Coord] = // [...]
@@ -1936,27 +1936,27 @@ val distances = coords1.cross(coords2) {
val dist = sqrt(pow(c1.x - c2.x, 2) + pow(c1.y - c2.y, 2))
(c1.id, c2.id, dist)
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class Euclid(CrossFunction):
def cross(self, c1, c2):
return (c1[0], c2[0], sqrt(pow(c1[1] - c2.[1], 2) + pow(c1[2] - c2[2], 2)))
distances = coords1.cross(coords2).using(Euclid())
-~~~
+{% endhighlight %}
#### Cross with Projection
A Cross transformation can also construct result tuples using a projection as shown here:
-~~~python
+{% highlight python %}
result = input1.cross(input2).projectFirst(1,0).projectSecond(0,1);
-~~~
+{% endhighlight %}
The field selection in a Cross projection works the same way as in the projection of Join results.
@@ -1970,7 +1970,7 @@ In order to guide the optimizer to pick the right execution strategy, you can hi
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<Integer, String>> input1 = // [...]
DataSet<Tuple2<Integer, String>> input2 = // [...]
@@ -1987,12 +1987,12 @@ DataSet<Tuple3<Integer, Integer, String>>
input1.crossWithHuge(input2)
// apply a projection (or any Cross function)
.projectFirst(0,1).projectSecond(1);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val input1: DataSet[(Int, String)] = // [...]
val input2: DataSet[(Int, String)] = // [...]
@@ -2002,19 +2002,19 @@ val result1 = input1.crossWithTiny(input2)
// hint that the second DataSet is very large
val result1 = input1.crossWithHuge(input2)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
#hint that the second DataSet is very small
result1 = input1.cross_with_tiny(input2)
#hint that the second DataSet is very large
result1 = input1.cross_with_huge(input2)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2033,7 +2033,7 @@ Similar to Reduce, GroupReduce, and Join, keys can be defined using the differen
The example shows how to group by Field Position Keys (Tuple DataSets only). You can do the same with Pojo-types and key expressions.
-~~~java
+{% highlight java %}
// Some CoGroupFunction definition
class MyCoGrouper
implements CoGroupFunction<Tuple2<String, Integer>, Tuple2<String, Double>, Double> {
@@ -2069,12 +2069,12 @@ DataSet<Double> output = iVals.coGroup(dVals)
.equalTo(0)
// apply CoGroup function on each pair of groups
.with(new MyCoGrouper());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val iVals: DataSet[(String, Int)] = // [...]
val dVals: DataSet[(String, Double)] = // [...]
@@ -2088,12 +2088,12 @@ val output = iVals.coGroup(dVals).where(0).equalTo(0) {
}
}
}
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
class CoGroup(CoGroupFunction):
def co_group(self, ivals, dvals, collector):
ints = dict()
@@ -2107,7 +2107,7 @@ val output = iVals.coGroup(dVals).where(0).equalTo(0) {
output = ivals.co_group(dvals).where(0).equal_to(0).using(CoGroup())
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2120,30 +2120,30 @@ Produces the union of two DataSets, which have to be of the same type. A union o
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<String, Integer>> vals1 = // [...]
DataSet<Tuple2<String, Integer>> vals2 = // [...]
DataSet<Tuple2<String, Integer>> vals3 = // [...]
DataSet<Tuple2<String, Integer>> unioned = vals1.union(vals2).union(vals3);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val vals1: DataSet[(String, Int)] = // [...]
val vals2: DataSet[(String, Int)] = // [...]
val vals3: DataSet[(String, Int)] = // [...]
val unioned = vals1.union(vals2).union(vals3)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
unioned = vals1.union(vals2).union(vals3)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2154,28 +2154,28 @@ Evenly rebalances the parallel partitions of a DataSet to eliminate data skew.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<String> in = // [...]
// rebalance DataSet and apply a Map transformation.
DataSet<Tuple2<String, String>> out = in.rebalance()
.map(new Mapper());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val in: DataSet[String] = // [...]
// rebalance DataSet and apply a Map transformation.
val out = in.rebalance().map { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2189,28 +2189,28 @@ Keys can be specified as position keys, expression keys, and key selector functi
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<String, Integer>> in = // [...]
// hash-partition DataSet by String value and apply a MapPartition transformation.
DataSet<Tuple2<String, String>> out = in.partitionByHash(0)
.mapPartition(new PartitionMapper());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val in: DataSet[(String, Int)] = // [...]
// hash-partition DataSet by String value and apply a MapPartition transformation.
val out = in.partitionByHash(0).mapPartition { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2223,28 +2223,28 @@ Keys can be specified as position keys, expression keys, and key selector functi
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<String, Integer>> in = // [...]
// range-partition DataSet by String value and apply a MapPartition transformation.
DataSet<Tuple2<String, String>> out = in.partitionByRange(0)
.mapPartition(new PartitionMapper());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val in: DataSet[(String, Int)] = // [...]
// range-partition DataSet by String value and apply a MapPartition transformation.
val out = in.partitionByRange(0).mapPartition { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2259,7 +2259,7 @@ Partitions can be sorted on multiple fields by chaining `sortPartition()` calls.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<String, Integer>> in = // [...]
// Locally sort partitions in ascending order on the second String field and
// in descending order on the first String field.
@@ -2267,12 +2267,12 @@ DataSet<Tuple2<String, Integer>> in = // [...]
DataSet<Tuple2<String, String>> out = in.sortPartition(1, Order.ASCENDING)
.sortPartition(0, Order.DESCENDING)
.mapPartition(new PartitionMapper());
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val in: DataSet[(String, Int)] = // [...]
// Locally sort partitions in ascending order on the second String field and
// in descending order on the first String field.
@@ -2280,14 +2280,14 @@ val in: DataSet[(String, Int)] = // [...]
val out = in.sortPartition(1, Order.ASCENDING)
.sortPartition(0, Order.DESCENDING)
.mapPartition { ... }
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
@@ -2299,7 +2299,7 @@ Returns the first n (arbitrary) elements of a DataSet. First-n can be applied on
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
DataSet<Tuple2<String, Integer>> in = // [...]
// Return the first five (arbitrary) elements of the DataSet
DataSet<Tuple2<String, Integer>> out1 = in.first(5);
@@ -2312,12 +2312,12 @@ DataSet<Tuple2<String, Integer>> out2 = in.groupBy(0)
DataSet<Tuple2<String, Integer>> out3 = in.groupBy(0)
.sortGroup(1, Order.ASCENDING)
.first(3);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val in: DataSet[(String, Int)] = // [...]
// Return the first five (arbitrary) elements of the DataSet
val out1 = in.first(5)
@@ -2327,14 +2327,14 @@ val out2 = in.groupBy(0).first(2)
// Return the first three elements of each String group ordered by the Integer field
val out3 = in.groupBy(0).sortGroup(1, Order.ASCENDING).first(3)
-~~~
+{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
-~~~python
+{% highlight python %}
Not supported.
-~~~
+{% endhighlight %}
</div>
</div>
diff --git a/docs/dev/batch/examples.md b/docs/dev/batch/examples.md
index a4b282688ee..90e372dfd60 100644
--- a/docs/dev/batch/examples.md
+++ b/docs/dev/batch/examples.md
@@ -44,17 +44,17 @@ Each binary release of Flink contains an `examples` directory with jar files for
To run the WordCount example, issue the following command:
-~~~bash
+{% highlight bash %}
./bin/flink run ./examples/batch/WordCount.jar
-~~~
+{% endhighlight %}
The other examples can be started in a similar way.
Note that many examples run without passing any arguments for them, by using build-in data. To run WordCount with real data, you have to pass the path to the data:
-~~~bash
+{% highlight bash %}
./bin/flink run ./examples/batch/WordCount.jar --input /path/to/some/text/data --output /path/to/result
-~~~
+{% endhighlight %}
Note that non-local file systems require a schema prefix, such as `hdfs://`.
@@ -65,7 +65,7 @@ WordCount is the "Hello World" of Big Data processing systems. It computes the f
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> text = env.readTextFile("/path/to/file");
@@ -95,14 +95,14 @@ public static class Tokenizer implements FlatMapFunction<String, Tuple2<String,
}
}
}
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java "WordCount example" %} implements the above described algorithm with input parameters: `--input <path> --output <path>`. As test data, any text file will do.
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val env = ExecutionEnvironment.getExecutionEnvironment
// get input data
@@ -114,7 +114,7 @@ val counts = text.flatMap { _.toLowerCase.split("\\W+") filter { _.nonEmpty } }
.sum(1)
counts.writeAsCsv(outputPath, "\n", " ")
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/wordcount/WordCount.scala "WordCount example" %} implements the above described algorithm with input parameters: `--input <path> --output <path>`. As test data, any text file will do.
@@ -131,7 +131,7 @@ In this simple example, PageRank is implemented with a [bulk iteration](iteratio
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// read the pages and initial ranks by parsing a CSV file
@@ -202,7 +202,7 @@ public static final class EpsilonFilter
return Math.abs(value.f0.f1 - value.f1.f1) > EPSILON;
}
}
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/graph/PageRank.java "PageRank program" %} implements the above example.
It requires the following parameters to run: `--pages <path> --links <path> --output <path> --numPages <n> --iterations <n>`.
@@ -210,7 +210,7 @@ It requires the following parameters to run: `--pages <path> --links <path> --ou
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// User-defined types
case class Link(sourceId: Long, targetId: Long)
case class Page(pageId: Long, rank: Double)
@@ -269,7 +269,7 @@ val result = finalRanks
// emit result
result.writeAsCsv(outputPath, "\n", " ")
-~~~
+{% endhighlight %}
he {% gh_link /flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/graph/PageRankBasic.scala "PageRank program" %} implements the above example.
It requires the following parameters to run: `--pages <path> --links <path> --output <path> --numPages <n> --iterations <n>`.
@@ -293,7 +293,7 @@ This implementation uses a [delta iteration](iterations.html): Vertices that hav
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// read vertex and edge data
DataSet<Long> vertices = getVertexDataSet(env);
DataSet<Tuple2<Long, Long>> edges = getEdgeDataSet(env).flatMap(new UndirectEdge());
@@ -365,14 +365,14 @@ public static final class ComponentIdFilter
}
}
}
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/graph/ConnectedComponents.java "ConnectedComponents program" %} implements the above example. It requires the following parameters to run: `--vertices <path> --edges <path> --output <path> --iterations <n>`.
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// set up execution environment
val env = ExecutionEnvironment.getExecutionEnvironment
@@ -408,7 +408,7 @@ val verticesWithComponents = vertices.iterateDelta(vertices, maxIterations, Arra
verticesWithComponents.writeAsCsv(outputPath, "\n", " ")
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/graph/ConnectedComponents.scala "ConnectedComponents program" %} implements the above example. It requires the following parameters to run: `--vertices <path> --edges <path> --output <path> --iterations <n>`.
</div>
@@ -426,7 +426,7 @@ The Relational Query example assumes two tables, one with `orders` and the other
The example implements the following SQL query.
-~~~sql
+{% highlight sql %}
SELECT l_orderkey, o_shippriority, sum(l_extendedprice) as revenue
FROM orders, lineitem
WHERE l_orderkey = o_orderkey
@@ -434,14 +434,14 @@ WHERE l_orderkey = o_orderkey
AND YEAR(o_orderdate) > 1993
AND o_orderpriority LIKE "5%"
GROUP BY l_orderkey, o_shippriority;
-~~~
+{% endhighlight %}
The Flink program, which implements the above query looks as follows.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// get orders data set: (orderkey, orderstatus, orderdate, orderpriority, shippriority)
DataSet<Tuple5<Integer, String, String, String, Integer>> orders = getOrdersDataSet(env);
// get lineitem data set: (orderkey, extendedprice)
@@ -484,7 +484,7 @@ DataSet<Tuple3<Integer, Integer, Double>> priceSums =
// emit result
priceSums.writeAsCsv(outputPath);
-~~~
+{% endhighlight %}
The {% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/TPCHQuery10.java "Relational Query program" %} implements the above query. It requires the following parameters to run: `--orders <path> --lineitem <path> --output <path>`.
@@ -503,19 +503,19 @@ Take the following steps to generate arbitrary large input files for the provide
1. Download and unpack DBGEN
2. Make a copy of *makefile.suite* called *Makefile* and perform the following changes:
-~~~bash
+{% highlight bash %}
DATABASE = DB2
MACHINE = LINUX
WORKLOAD = TPCH
CC = gcc
-~~~
+{% endhighlight %}
1. Build DBGEN using *make*
2. Generate lineitem and orders relations using dbgen. A scale factor
(-s) of 1 results in a generated data set with about 1 GB size.
-~~~bash
+{% highlight bash %}
./dbgen -T o -s 1
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/dev/batch/fault_tolerance.md b/docs/dev/batch/fault_tolerance.md
index f4037910257..ee1e7973188 100644
--- a/docs/dev/batch/fault_tolerance.md
+++ b/docs/dev/batch/fault_tolerance.md
@@ -59,9 +59,9 @@ env.setNumberOfExecutionRetries(3)
You can also define default values for the number of execution retries and the retry delay in the `flink-conf.yaml`:
-~~~
+{% highlight yaml %}
execution-retries.default: 3
-~~~
+{% endhighlight %}
Retry Delays
@@ -91,8 +91,8 @@ env.getConfig.setExecutionRetryDelay(5000) // 5000 milliseconds delay
You can also define the default value for the retry delay in the `flink-conf.yaml`:
-~~~
+{% highlight yaml %}
execution-retries.delay: 10 s
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/dev/batch/hadoop_compatibility.md b/docs/dev/batch/hadoop_compatibility.md
index 9f1478a243e..c665b5d99a0 100644
--- a/docs/dev/batch/hadoop_compatibility.md
+++ b/docs/dev/batch/hadoop_compatibility.md
@@ -56,13 +56,13 @@ package.
Add the following dependency to your `pom.xml` if you want to reuse Mappers
and Reducers.
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-compatibility{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
</dependency>
-~~~
+{% endhighlight %}
### Using Hadoop Data Types
@@ -73,11 +73,14 @@ if you only want to use your Hadoop data types. See the
### Using Hadoop InputFormats
-Hadoop input formats can be used to create a data source by using
-one of the methods `readHadoopFile` or `createHadoopInput` of the
-`ExecutionEnvironment`. The former is used for input formats derived
+To use Hadoop `InputFormats` with Flink the format must first be wrapped
+using either `readHadoopFile` or `createHadoopInput` of the
+`HadoopInputs` utilty class.
+The former is used for input formats derived
from `FileInputFormat` while the latter has to be used for general purpose
input formats.
+The resulting `InputFormat` can be used to create a data source by using
+`ExecutionEnvironmen#createInput`.
The resulting `DataSet` contains 2-tuples where the first field
is the key and the second field is the value retrieved from the Hadoop
@@ -88,28 +91,30 @@ The following example shows how to use Hadoop's `TextInputFormat`.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<Tuple2<LongWritable, Text>> input =
- env.readHadoopFile(new TextInputFormat(), LongWritable.class, Text.class, textPath);
+ env.createInput(HadoopInputs.readHadoopFile(new TextInputFormat(),
+ LongWritable.class, Text.class, textPath));
// Do something with the data.
[...]
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val env = ExecutionEnvironment.getExecutionEnvironment
val input: DataSet[(LongWritable, Text)] =
- env.readHadoopFile(new TextInputFormat, classOf[LongWritable], classOf[Text], textPath)
+ env.createInput(HadoopInputs.readHadoopFile(
+ new TextInputFormat, classOf[LongWritable], classOf[Text], textPath))
// Do something with the data.
[...]
-~~~
+{% endhighlight %}
</div>
@@ -128,7 +133,7 @@ The following example shows how to use Hadoop's `TextOutputFormat`.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
// Obtain the result we want to emit
DataSet<Tuple2<Text, IntWritable>> hadoopResult = [...]
@@ -144,12 +149,12 @@ TextOutputFormat.setOutputPath(job, new Path(outputPath));
// Emit data using the Hadoop TextOutputFormat.
hadoopResult.output(hadoopOF);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
// Obtain your result to emit.
val hadoopResult: DataSet[(Text, IntWritable)] = [...]
@@ -163,7 +168,7 @@ FileOutputFormat.setOutputPath(hadoopOF.getJobConf, new Path(resultPath))
hadoopResult.output(hadoopOF)
-~~~
+{% endhighlight %}
</div>
@@ -185,7 +190,7 @@ and can be used as regular Flink [FlatMapFunctions](dataset_transformations.html
The following example shows how to use Hadoop `Mapper` and `Reducer` functions.
-~~~java
+{% highlight java %}
// Obtain data to process somehow.
DataSet<Tuple2<Text, LongWritable>> text = [...]
@@ -199,7 +204,7 @@ DataSet<Tuple2<Text, LongWritable>> result = text
.reduceGroup(new HadoopReduceCombineFunction<Text, LongWritable, Text, LongWritable>(
new Counter(), new Counter()
));
-~~~
+{% endhighlight %}
**Please note:** The Reducer wrapper works on groups as defined by Flink's [groupBy()](dataset_transformations.html#transformations-on-grouped-dataset) operation. It does not consider any custom partitioners, sort or grouping comparators you might have set in the `JobConf`.
@@ -207,7 +212,7 @@ DataSet<Tuple2<Text, LongWritable>> result = text
The following example shows a complete WordCount implementation using Hadoop data types, Input- and OutputFormats, and Mapper and Reducer implementations.
-~~~java
+{% highlight java %}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// Set up the Hadoop TextInputFormat.
@@ -245,6 +250,6 @@ result.output(hadoopOF);
// Execute Program
env.execute("Hadoop WordCount");
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/dev/batch/index.md b/docs/dev/batch/index.md
index 24b33902f1a..c624fce8954 100644
--- a/docs/dev/batch/index.md
+++ b/docs/dev/batch/index.md
@@ -824,9 +824,6 @@ File-based:
- `readFileOfPrimitives(path, delimiter, Class)` / `PrimitiveInputFormat` - Parses files of new-line (or another char sequence)
delimited primitive data types such as `String` or `Integer` using the given delimiter.
-- `readHadoopFile(FileInputFormat, Key, Value, path)` / `FileInputFormat` - Creates a JobConf and reads file from the specified
- path with the specified FileInputFormat, Key class and Value class and returns them as Tuple2<Key, Value>.
-
- `readSequenceFile(Key, Value, path)` / `SequenceFileInputFormat` - Creates a JobConf and reads file from the specified path with
type SequenceFileInputFormat, Key class and Value class and returns them as Tuple2<Key, Value>.
@@ -878,11 +875,6 @@ DataSet<Tuple2<String, Double>> csvInput = env.readCsvFile("hdfs:///the/CSV/file
DataSet<Person>> csvInput = env.readCsvFile("hdfs:///the/CSV/file")
.pojoType(Person.class, "name", "age", "zipcode");
-
-// read a file from the specified path of type TextInputFormat
-DataSet<Tuple2<LongWritable, Text>> tuples =
- env.readHadoopFile(new TextInputFormat(), LongWritable.class, Text.class, "hdfs://nnHost:nnPort/path/to/file");
-
// read a file from the specified path of type SequenceFileInputFormat
DataSet<Tuple2<IntWritable, Text>> tuples =
env.readSequenceFile(IntWritable.class, Text.class, "hdfs://nnHost:nnPort/path/to/file");
@@ -974,9 +966,6 @@ File-based:
- `readFileOfPrimitives(path, delimiter)` / `PrimitiveInputFormat` - Parses files of new-line (or another char sequence)
delimited primitive data types such as `String` or `Integer` using the given delimiter.
-- `readHadoopFile(FileInputFormat, Key, Value, path)` / `FileInputFormat` - Creates a JobConf and reads file from the specified
- path with the specified FileInputFormat, Key class and Value class and returns them as Tuple2<Key, Value>.
-
- `readSequenceFile(Key, Value, path)` / `SequenceFileInputFormat` - Creates a JobConf and reads file from the specified path with
type SequenceFileInputFormat, Key class and Value class and returns them as Tuple2<Key, Value>.
@@ -1039,10 +1028,6 @@ val values = env.fromElements("Foo", "bar", "foobar", "fubar")
// generate a number sequence
val numbers = env.generateSequence(1, 10000000)
-// read a file from the specified path of type TextInputFormat
-val tuples = env.readHadoopFile(new TextInputFormat, classOf[LongWritable],
- classOf[Text], "hdfs://nnHost:nnPort/path/to/file")
-
// read a file from the specified path of type SequenceFileInputFormat
val tuples = env.readSequenceFile(classOf[IntWritable], classOf[Text],
"hdfs://nnHost:nnPort/path/to/file")
diff --git a/docs/dev/batch/iterations.md b/docs/dev/batch/iterations.md
index f5b65ac8d79..be0ebb96480 100644
--- a/docs/dev/batch/iterations.md
+++ b/docs/dev/batch/iterations.md
@@ -102,7 +102,7 @@ There are multiple options to specify **termination conditions** for an iteratio
You can also think about the iterate operator in pseudo-code:
-~~~java
+{% highlight java %}
IterationState state = getInitialState();
while (!terminationCriterion()) {
@@ -110,7 +110,7 @@ while (!terminationCriterion()) {
}
setFinalState(state);
-~~~
+{% endhighlight %}
<div class="panel panel-default">
<div class="panel-body">
@@ -130,14 +130,14 @@ In the following example, we **iteratively increment a set numbers**:
3. **Next Partial Solution**: The output of the step function will be the output of the map operator, i.e. records with incremented integers.
4. **Iteration Result**: After ten iterations, the initial numbers will have been incremented ten times, resulting in integers `11` to `15`.
-~~~
+{% highlight plain %}
// 1st 2nd 10th
map(1) -> 2 map(2) -> 3 ... map(10) -> 11
map(2) -> 3 map(3) -> 4 ... map(11) -> 12
map(3) -> 4 map(4) -> 5 ... map(12) -> 13
map(4) -> 5 map(5) -> 6 ... map(13) -> 14
map(5) -> 6 map(6) -> 7 ... map(14) -> 15
-~~~
+{% endhighlight %}
Note that **1**, **2**, and **4** can be arbitrary data flows.
@@ -162,7 +162,7 @@ The default **termination condition** for delta iterations is specified by the *
You can also think about the iterate operator in pseudo-code:
-~~~java
+{% highlight java %}
IterationState workset = getInitialState();
IterationState solution = getInitialSolution();
@@ -173,7 +173,7 @@ while (!terminationCriterion()) {
}
setFinalState(solution);
-~~~
+{% endhighlight %}
<div class="panel panel-default">
<div class="panel-body">
diff --git a/docs/dev/batch/python.md b/docs/dev/batch/python.md
index 22111027ee8..486aa18c9b7 100644
--- a/docs/dev/batch/python.md
+++ b/docs/dev/batch/python.md
@@ -136,10 +136,7 @@ The first two do as the name suggests.
Please refer to [Data Sinks](#data-sinks) for more information on writing to files.
Once you specified the complete program you need to call `execute` on
-the `Environment`. This will either execute on your local machine or submit your program
-for execution on a cluster, depending on how Flink was started. You can force
-a local execution by using `execute(local=True)`.
-
+the `Environment`. This will submit your program for execution on a cluster.
{% top %}
Project setup
@@ -159,8 +156,7 @@ Lazy Evaluation
All Flink programs are executed lazily: When the program's main method is executed, the data loading
and transformations do not happen directly. Rather, each operation is created and added to the
program's plan. The operations are actually executed when one of the `execute()` methods is invoked
-on the Environment object. Whether the program is executed locally or on a cluster depends
-on the environment of the program.
+on the Environment object.
The lazy evaluation lets you construct sophisticated programs that Flink executes as one
holistically planned unit.
diff --git a/docs/dev/best_practices.md b/docs/dev/best_practices.md
index 24e7091974d..e8dee306a55 100644
--- a/docs/dev/best_practices.md
+++ b/docs/dev/best_practices.md
@@ -149,20 +149,20 @@ Also, POJOs can be used to give large `Tuple`-types a name.
Instead of using:
-~~~java
+{% highlight java %}
Tuple11<String, String, ..., String> var = new ...;
-~~~
+{% endhighlight %}
It is much easier to create a custom type extending from the large Tuple type.
-~~~java
+{% highlight java %}
CustomType var = new ...;
public static class CustomType extends Tuple11<String, String, ..., String> {
// constructor matching super
}
-~~~
+{% endhighlight %}
## Using Logback instead of Log4j
diff --git a/docs/dev/cluster_execution.md b/docs/dev/cluster_execution.md
index f1d84e1b67b..2b290549ff7 100644
--- a/docs/dev/cluster_execution.md
+++ b/docs/dev/cluster_execution.md
@@ -47,22 +47,22 @@ execute the program.
If you are developing your program as a Maven project, you have to add the
`flink-clients` module using this dependency:
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
<version>{{ site.version }}</version>
</dependency>
-~~~
+{% endhighlight %}
### Example
The following illustrates the use of the `RemoteEnvironment`:
-~~~java
+{% highlight java %}
public static void main(String[] args) throws Exception {
ExecutionEnvironment env = ExecutionEnvironment
- .createRemoteEnvironment("flink-master", 6123, "/home/user/udfs.jar");
+ .createRemoteEnvironment("flink-master", 8081, "/home/user/udfs.jar");
DataSet<String> data = env.readTextFile("hdfs://path/to/file");
@@ -76,7 +76,7 @@ public static void main(String[] args) throws Exception {
env.execute();
}
-~~~
+{% endhighlight %}
Note that the program contains custom user code and hence requires a JAR file with
the classes of the code attached. The constructor of the remote environment
diff --git a/docs/dev/connectors/filesystem_sink.md b/docs/dev/connectors/filesystem_sink.md
index 4e1f68a9f3a..af1349d6665 100644
--- a/docs/dev/connectors/filesystem_sink.md
+++ b/docs/dev/connectors/filesystem_sink.md
@@ -89,8 +89,13 @@ and write them to part files, separated by newline. To specify a custom writer u
on a `BucketingSink`. If you want to write Hadoop SequenceFiles you can use the provided
`SequenceFileWriter` which can also be configured to use compression.
-The last configuration option is the batch size. This specifies when a part file should be closed
-and a new one started. (The default part file size is 384 MB).
+There are two configuration options that specify when a part file should be closed
+and a new one started:
+
+* By setting a batch size (The default part file size is 384 MB)
+* By setting a batch roll over time interval (The default roll over interval is `Long.MAX_VALUE`)
+
+A new part file is started when either of these two conditions is satisfied.
Example:
@@ -103,6 +108,7 @@ BucketingSink<String> sink = new BucketingSink<String>("/base/path");
sink.setBucketer(new DateTimeBucketer<String>("yyyy-MM-dd--HHmm"));
sink.setWriter(new SequenceFileWriter<IntWritable, Text>());
sink.setBatchSize(1024 * 1024 * 400); // this is 400 MB,
+sink.setBatchRolloverInterval(20 * 60 * 1000); // this is 20 mins
input.addSink(sink);
@@ -116,6 +122,7 @@ val sink = new BucketingSink[String]("/base/path")
sink.setBucketer(new DateTimeBucketer[String]("yyyy-MM-dd--HHmm"))
sink.setWriter(new SequenceFileWriter[IntWritable, Text]())
sink.setBatchSize(1024 * 1024 * 400) // this is 400 MB,
+sink.setBatchRolloverInterval(20 * 60 * 1000); // this is 20 mins
input.addSink(sink)
@@ -125,13 +132,13 @@ input.addSink(sink)
This will create a sink that writes to bucket files that follow this schema:
-```
+{% highlight plain %}
/base/path/{date-time}/part-{parallel-task}-{count}
-```
+{% endhighlight %}
Where `date-time` is the string that we get from the date/time format, `parallel-task` is the index
-of the parallel sink instance and `count` is the running number of part files that where created
-because of the batch size.
+of the parallel sink instance and `count` is the running number of part files that were created
+because of the batch size or batch roll over interval.
For in-depth information, please refer to the JavaDoc for
[BucketingSink](http://flink.apache.org/docs/latest/api/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.html).
diff --git a/docs/dev/connectors/kinesis.md b/docs/dev/connectors/kinesis.md
index 4ee9c6b03d4..5e0b2f3aa82 100644
--- a/docs/dev/connectors/kinesis.md
+++ b/docs/dev/connectors/kinesis.md
@@ -52,6 +52,15 @@ cd flink-dist
mvn clean install -Pinclude-kinesis -DskipTests
{% endhighlight %}
+<span class="label label-danger">Attention</span> For Flink versions 1.4.2 and below, the KPL client version
+used by default in the Kinesis connectors, KPL 0.12.5, is no longer supported by AWS Kinesis Streams
+(see [here](https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-upgrades.html)).
+This means that when building the Kinesis connector, you will need to specify a higher version KPL client (above 0.12.6)
+in order for the Flink Kinesis Producer to work. You can do this by specifying the preferred version via the
+`aws.kinesis-kpl.version` property, like so:
+{% highlight bash %}
+mvn clean install -Pinclude-kinesis -Daws.kinesis-kpl.version=0.12.6 -DskipTests
+{% endhighlight %}
The streaming connectors are not part of the binary distribution. See how to link with them for cluster
execution [here]({{site.baseurl}}/dev/linking.html).
@@ -104,7 +113,7 @@ The above is a simple example of using the consumer. Configuration for the consu
instance, the configuration keys for which can be found in `ConsumerConfigConstants`. The example
demonstrates consuming a single Kinesis stream in the AWS region "us-east-1". The AWS credentials are supplied using the basic method in which
the AWS access key ID and secret access key are directly supplied in the configuration (other options are setting
-`ConsumerConfigConstants.AWS_CREDENTIALS_PROVIDER` to `ENV_VAR`, `SYS_PROP`, `PROFILE`, and `AUTO`). Also, data is being consumed
+`ConsumerConfigConstants.AWS_CREDENTIALS_PROVIDER` to `ENV_VAR`, `SYS_PROP`, `PROFILE`, `ASSUME_ROLE`, and `AUTO`). Also, data is being consumed
from the newest position in the Kinesis stream (the other option will be setting `ConsumerConfigConstants.STREAM_INITIAL_POSITION`
to `TRIM_HORIZON`, which lets the consumer start reading the Kinesis stream from the earliest record possible).
@@ -286,6 +295,8 @@ producerConfig.put("RecordTtl", "30000");
producerConfig.put("RequestTimeout", "6000");
producerConfig.put("ThreadPoolSize", "15");
+// Disable Aggregation if it's not supported by a consumer
+// producerConfig.put("AggregationEnabled", "false");
// Switch KinesisProducer's threading model
// producerConfig.put("ThreadingModel", "PER_REQUEST");
@@ -312,6 +323,8 @@ producerConfig.put("RecordTtl", "30000")
producerConfig.put("RequestTimeout", "6000")
producerConfig.put("ThreadPoolSize", "15")
+// Disable Aggregation if it's not supported by a consumer
+// producerConfig.put("AggregationEnabled", "false")
// Switch KinesisProducer's threading model
// producerConfig.put("ThreadingModel", "PER_REQUEST")
@@ -340,6 +353,36 @@ Since Flink 1.4.0, `FlinkKinesisProducer` switches its default underlying KPL fr
Users can still switch back to one-thread-per-request mode by setting a key-value pair of `ThreadingModel` and `PER_REQUEST` in `java.util.Properties`, as shown in the code commented out in above example.
+### Backpressure
+
+By default, `FlinkKinesisProducer` does not backpressure. Instead, records that
+cannot be sent because of the rate restriction of 1 MB per second per shard are
+buffered in an unbounded queue and dropped when their `RecordTtl` expires.
+
+To avoid data loss, you can enable backpressuring by restricting the size of the
+internal queue:
+
+```
+// 200 Bytes per record, 1 shard
+kinesis.setQueueLimit(500);
+```
+
+The value for `queueLimit` depends on the expected record size. To choose a good
+value, consider that Kinesis is rate-limited to 1MB per second per shard. If
+less than one second's worth of records is buffered, then the queue may not be
+able to operate at full capacity. With the default `RecordMaxBufferedTime` of
+100ms, a queue size of 100kB per shard should be sufficient. The `queueLimit`
+can then be computed via
+
+```
+queue limit = (number of shards * queue size per shard) / record size
+```
+
+E.g. for 200Bytes per record and 8 shards, a queue limit of 4000 is a good
+starting point. If the queue size limits throughput (below 1MB per second per
+shard), try increasing the queue limit slightly.
+
+
## Using Non-AWS Kinesis Endpoints for Testing
It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as
diff --git a/docs/dev/datastream_api.md b/docs/dev/datastream_api.md
index 6f30f29c231..b2c8cb81c2c 100644
--- a/docs/dev/datastream_api.md
+++ b/docs/dev/datastream_api.md
@@ -125,9 +125,9 @@ object WindowWordCount {
To run the example program, start the input stream with netcat first from a terminal:
-~~~bash
+{% highlight bash %}
nc -lk 9999
-~~~
+{% endhighlight %}
Just type some words hitting return for a new word. These will be the input to the
word count program. If you want to see counts greater than 1, type the same word again and again within
@@ -453,9 +453,6 @@ The `StreamExecutionEnvironment` contains the `ExecutionConfig` which allows to
Please refer to [execution configuration]({{ site.baseurl }}/dev/execution_configuration.html)
for an explanation of most parameters. These parameters pertain specifically to the DataStream API:
-- `enableTimestamps()` / **`disableTimestamps()`**: Attach a timestamp to each event emitted from a source.
- `areTimestampsEnabled()` returns the current value.
-
- `setAutoWatermarkInterval(long milliseconds)`: Set the interval for automatic watermark emission. You can
get the current value with `long getAutoWatermarkInterval()`
diff --git a/docs/dev/event_time.md b/docs/dev/event_time.md
index a3e697d8f7e..886bf22e8b2 100644
--- a/docs/dev/event_time.md
+++ b/docs/dev/event_time.md
@@ -35,31 +35,34 @@ Flink supports different notions of *time* in streaming programs.
respective operation.
When a streaming program runs on processing time, all time-based operations (like time windows) will
- use the system clock of the machines that run the respective operator. For example, an hourly
+ use the system clock of the machines that run the respective operator. An hourly
processing time window will include all records that arrived at a specific operator between the
- times when the system clock indicated the full hour.
+ times when the system clock indicated the full hour. For example, if an application
+ begins running at 9:15am, the first hourly processing time window will include events
+ processed between 9:15am and 10:00am, the next window will include events processed between 10:00am and 11:00am, and so on.
Processing time is the simplest notion of time and requires no coordination between streams and machines.
It provides the best performance and the lowest latency. However, in distributed and asynchronous
environments processing time does not provide determinism, because it is susceptible to the speed at which
- records arrive in the system (for example from the message queue), and to the speed at which the
- records flow between operators inside the system.
+ records arrive in the system (for example from the message queue), to the speed at which the
+ records flow between operators inside the system, and to outages (scheduled, or otherwise).
- **Event time:** Event time is the time that each individual event occurred on its producing device.
- This time is typically embedded within the records before they enter Flink and that *event timestamp*
- can be extracted from the record. An hourly event time window will contain all records that carry an
- event timestamp that falls into that hour, regardless of when the records arrive, and in what order
- they arrive.
-
- Event time gives correct results even on out-of-order events, late events, or on replays
- of data from backups or persistent logs. In event time, the progress of time depends on the data,
+ This time is typically embedded within the records before they enter Flink, and that *event timestamp*
+ can be extracted from each record. In event time, the progress of time depends on the data,
not on any wall clocks. Event time programs must specify how to generate *Event Time Watermarks*,
- which is the mechanism that signals progress in event time. The mechanism is
- described below.
+ which is the mechanism that signals progress in event time. This watermarking mechanism is
+ described in a later section, [below](#event-time-and-watermarks).
+
+ In a perfect world, event time processing would yield completely consistent and deterministic results, regardless of when events arrive, or their ordering.
+ However, unless the events are known to arrive in-order (by timestamp), event time processing incurs some latency while waiting for out-of-order events. As it is only possible to wait for a finite period of time, this places a limit on how deterministic event time applications can be.
+
+ Assuming all of the data has arrived, event time operations will behave as expected, and produce correct and consistent results even when working with out-of-order or late events, or when reprocessing historic data. For example, an hourly event time window will contain all records
+ that carry an event timestamp that falls into that hour, regardless of the order in which they arrive, or when they are processed. (See the section on [late events](#late-elements) for more information.)
- Event time processing often incurs a certain latency, due to its nature of waiting a certain time for
- late events and out-of-order events. Because of that, event time programs are often combined with
- *processing time* operations.
+
+
+ Note that sometimes when event time programs are processing live data in real-time, they will use some *processing time* operations in order to guarantee that they are progressing in a timely fashion.
- **Ingestion time:** Ingestion time is the time that events enter Flink. At the source operator each
record gets the source's current time as a timestamp, and time-based operations (like time windows)
@@ -194,6 +197,8 @@ The figure below shows an example of events and watermarks flowing through paral
<img src="{{ site.baseurl }}/fig/parallel_streams_watermarks.svg" alt="Parallel data streams and operators with events and watermarks" class="center" width="80%" />
+Note that the Kafka source supports per-partition watermarking, which you can read more about [here]({{ site.baseurl }}/dev/event_timestamps_watermarks.html#timestamps-per-kafka-partition).
+
## Late Elements
@@ -201,7 +206,7 @@ It is possible that certain elements will violate the watermark condition, meani
more elements with timestamp *t' <= t* will occur. In fact, in many real world setups, certain elements can be arbitrarily
delayed, making it impossible to specify a time by which all elements of a certain event timestamp will have occurred.
Furthermore, even if the lateness can be bounded, delaying the watermarks by too much is often not desirable, because it
-causes too much delay in the evaluation of the event time windows.
+causes too much delay in the evaluation of event time windows.
For this reason, streaming programs may explicitly expect some *late* elements. Late elements are elements that
arrive after the system's event time clock (as signaled by the watermarks) has already passed the time of the late element's
diff --git a/docs/dev/event_timestamps_watermarks.md b/docs/dev/event_timestamps_watermarks.md
index 44ecf54244b..cb1c5d4e021 100644
--- a/docs/dev/event_timestamps_watermarks.md
+++ b/docs/dev/event_timestamps_watermarks.md
@@ -58,14 +58,14 @@ progress in event time.
There are two ways to assign timestamps and generate watermarks:
1. Directly in the data stream source
- 2. Via a timestamp assigner / watermark generator: in Flink timestamp assigners also define the watermarks to be emitted
+ 2. Via a timestamp assigner / watermark generator: in Flink, timestamp assigners also define the watermarks to be emitted
<span class="label label-danger">Attention</span> Both timestamps and watermarks are specified as
milliseconds since the Java epoch of 1970-01-01T00:00:00Z.
### Source Functions with Timestamps and Watermarks
-Stream sources can also directly assign timestamps to the elements they produce, and they can also emit watermarks.
+Stream sources can directly assign timestamps to the elements they produce, and they can also emit watermarks.
When this is done, no timestamp assigner is needed.
Note that if a timestamp assigner is used, any timestamps and watermarks provided by the source will be overwritten.
@@ -180,7 +180,7 @@ The interval (every *n* milliseconds) in which the watermark will be generated i
called each time, and a new watermark will be emitted if the returned watermark is non-null and larger than the previous
watermark.
-Two simple examples of timestamp assigners with periodic watermark generation are below.
+Here we show two simple examples of timestamp assigners that use periodic watermark generation. Note that Flink ships with a `BoundedOutOfOrdernessTimestampExtractor` similar to the `BoundedOutOfOrdernessGenerator` shown below, which you can read about [here]({{ site.baseurl }}/dev/event_timestamp_extractors.html#assigners-allowing-a-fixed-amount-of-lateness).
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -242,7 +242,7 @@ class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks[MyEv
val maxOutOfOrderness = 3500L // 3.5 seconds
- var currentMaxTimestamp: Long
+ var currentMaxTimestamp: Long = _
override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = {
val timestamp = element.getCreationTime()
diff --git a/docs/dev/execution_configuration.md b/docs/dev/execution_configuration.md
index 8fe1b6376e7..f0103b0f39f 100644
--- a/docs/dev/execution_configuration.md
+++ b/docs/dev/execution_configuration.md
@@ -55,7 +55,7 @@ With the closure cleaner disabled, it might happen that an anonymous user functi
- `getExecutionMode()` / `setExecutionMode()`. The default execution mode is PIPELINED. Sets the execution mode to execute the program. The execution mode defines whether data exchanges are performed in a batch or on a pipelined manner.
-- `enableForceKryo()` / **`disableForceKryo`**. Kryo is not forced by default. Forces the GenericTypeInformation to use the Kryo serializer for POJOS even though we could analyze them as a POJO. In some cases this might be preferable. For example, when Flink's internal serializers fail to handle a POJO properly.
+- `enableForceKryo()` / **`disableForceKryo`**. Kryo is not forced by default. Forces the GenericTypeInformation to use the Kryo serializer for POJOs even though we could analyze them as a POJO. In some cases this might be preferable. For example, when Flink's internal serializers fail to handle a POJO properly.
- `enableForceAvro()` / **`disableForceAvro()`**. Avro is not forced by default. Forces the Flink AvroTypeInformation to use the Avro serializer instead of Kryo for serializing Avro POJOs.
diff --git a/docs/dev/java8.md b/docs/dev/java8.md
deleted file mode 100644
index 49498336478..00000000000
--- a/docs/dev/java8.md
+++ /dev/null
@@ -1,198 +0,0 @@
----
-title: "Java 8"
-nav-parent_id: api-concepts
-nav-pos: 20
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Java 8 introduces several new language features designed for faster and clearer coding. With the most important feature,
-the so-called "Lambda Expressions", Java 8 opens the door to functional programming. Lambda Expressions allow for implementing and
-passing functions in a straightforward way without having to declare additional (anonymous) classes.
-
-The newest version of Flink supports the usage of Lambda Expressions for all operators of the Java API.
-This document shows how to use Lambda Expressions and describes current limitations. For a general introduction to the
-Flink API, please refer to the [Programming Guide]({{ site.baseurl }}/dev/api_concepts.html)
-
-* TOC
-{:toc}
-
-### Examples
-
-The following example illustrates how to implement a simple, inline `map()` function that squares its input using a Lambda Expression.
-The types of input `i` and output parameters of the `map()` function need not to be declared as they are inferred by the Java 8 compiler.
-
-~~~java
-env.fromElements(1, 2, 3)
-// returns the squared i
-.map(i -> i*i)
-.print();
-~~~
-
-The next two examples show different implementations of a function that uses a `Collector` for output.
-Functions, such as `flatMap()`, require an output type (in this case `String`) to be defined for the `Collector` in order to be type-safe.
-If the `Collector` type can not be inferred from the surrounding context, it needs to be declared in the Lambda Expression's parameter list manually.
-Otherwise the output will be treated as type `Object` which can lead to undesired behaviour.
-
-~~~java
-DataSet<Integer> input = env.fromElements(1, 2, 3);
-
-// collector type must be declared
-input.flatMap((Integer number, Collector<String> out) -> {
- StringBuilder builder = new StringBuilder();
- for(int i = 0; i < number; i++) {
- builder.append("a");
- out.collect(builder.toString());
- }
-})
-// returns (on separate lines) "a", "a", "aa", "a", "aa", "aaa"
-.print();
-~~~
-
-~~~java
-DataSet<Integer> input = env.fromElements(1, 2, 3);
-
-// collector type must not be declared, it is inferred from the type of the dataset
-DataSet<String> manyALetters = input.flatMap((number, out) -> {
- StringBuilder builder = new StringBuilder();
- for(int i = 0; i < number; i++) {
- builder.append("a");
- out.collect(builder.toString());
- }
-});
-
-// returns (on separate lines) "a", "a", "aa", "a", "aa", "aaa"
-manyALetters.print();
-~~~
-
-The following code demonstrates a word count which makes extensive use of Lambda Expressions.
-
-~~~java
-DataSet<String> input = env.fromElements("Please count", "the words", "but not this");
-
-// filter out strings that contain "not"
-input.filter(line -> !line.contains("not"))
-// split each line by space
-.map(line -> line.split(" "))
-// emit a pair <word,1> for each array element
-.flatMap((String[] wordArray, Collector<Tuple2<String, Integer>> out)
- -> Arrays.stream(wordArray).forEach(t -> out.collect(new Tuple2<>(t, 1)))
- )
-// group and sum up
-.groupBy(0).sum(1)
-// print
-.print();
-~~~
-
-### Compiler Limitations
-Currently, Flink only supports jobs containing Lambda Expressions completely if they are **compiled with the Eclipse JDT compiler contained in Eclipse Luna 4.4.2 (and above)**.
-
-Only the Eclipse JDT compiler preserves the generic type information necessary to use the entire Lambda Expressions feature type-safely.
-Other compilers such as the OpenJDK's and Oracle JDK's `javac` throw away all generic parameters related to Lambda Expressions. This means that types such as `Tuple2<String, Integer>` or `Collector<String>` declared as a Lambda function input or output parameter will be pruned to `Tuple2` or `Collector` in the compiled `.class` files, which is too little information for the Flink compiler.
-
-How to compile a Flink job that contains Lambda Expressions with the JDT compiler will be covered in the next section.
-
-However, it is possible to implement functions such as `map()` or `filter()` with Lambda Expressions in Java 8 compilers other than the Eclipse JDT compiler as long as the function has no `Collector`s or `Iterable`s *and* only if the function handles unparameterized types such as `Integer`, `Long`, `String`, `MyOwnClass` (types without Generics!).
-
-#### Compile Flink jobs with the Eclipse JDT compiler and Maven
-
-If you are using the Eclipse IDE, you can run and debug your Flink code within the IDE without any problems after some configuration steps. The Eclipse IDE by default compiles its Java sources with the Eclipse JDT compiler. The next section describes how to configure the Eclipse IDE.
-
-If you are using a different IDE such as IntelliJ IDEA or you want to package your Jar-File with Maven to run your job on a cluster, you need to modify your project's `pom.xml` file and build your program with Maven. The [quickstart]({{site.baseurl}}/quickstart/setup_quickstart.html) contains preconfigured Maven projects which can be used for new projects or as a reference. Uncomment the mentioned lines in your generated quickstart `pom.xml` file if you want to use Java 8 with Lambda Expressions.
-
-Alternatively, you can manually insert the following lines to your Maven `pom.xml` file. Maven will then use the Eclipse JDT compiler for compilation.
-
-~~~xml
-<!-- put these lines under "project/build/pluginManagement/plugins" of your pom.xml -->
-
-<plugin>
- <!-- Use compiler plugin with tycho as the adapter to the JDT compiler. -->
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>1.8</source>
- <target>1.8</target>
- <compilerId>jdt</compilerId>
- </configuration>
- <dependencies>
- <!-- This dependency provides the implementation of compiler "jdt": -->
- <dependency>
- <groupId>org.eclipse.tycho</groupId>
- <artifactId>tycho-compiler-jdt</artifactId>
- <version>0.21.0</version>
- </dependency>
- </dependencies>
-</plugin>
-~~~
-
-If you are using Eclipse for development, the m2e plugin might complain about the inserted lines above and marks your `pom.xml` as invalid. If so, insert the following lines to your `pom.xml`.
-
-~~~xml
-<!-- put these lines under "project/build/pluginManagement/plugins/plugin[groupId="org.eclipse.m2e", artifactId="lifecycle-mapping"]/configuration/lifecycleMappingMetadata/pluginExecutions" of your pom.xml -->
-
-<pluginExecution>
- <pluginExecutionFilter>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <versionRange>[3.1,)</versionRange>
- <goals>
- <goal>testCompile</goal>
- <goal>compile</goal>
- </goals>
- </pluginExecutionFilter>
- <action>
- <ignore></ignore>
- </action>
-</pluginExecution>
-~~~
-
-#### Run and debug Flink jobs within the Eclipse IDE
-
-First of all, make sure you are running a current version of Eclipse IDE (4.4.2 or later). Also make sure that you have a Java 8 Runtime Environment (JRE) installed in Eclipse IDE (`Window` -> `Preferences` -> `Java` -> `Installed JREs`).
-
-Create/Import your Eclipse project.
-
-If you are using Maven, you also need to change the Java version in your `pom.xml` for the `maven-compiler-plugin`. Otherwise right click the `JRE System Library` section of your project and open the `Properties` window in order to switch to a Java 8 JRE (or above) that supports Lambda Expressions.
-
-The Eclipse JDT compiler needs a special compiler flag in order to store type information in `.class` files. Open the JDT configuration file at `{project directory}/.settings/org.eclipse.jdt.core.prefs` with your favorite text editor and add the following line:
-
-~~~
-org.eclipse.jdt.core.compiler.codegen.lambda.genericSignature=generate
-~~~
-
-If not already done, also modify the Java versions of the following properties to `1.8` (or above):
-
-~~~
-org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8
-org.eclipse.jdt.core.compiler.compliance=1.8
-org.eclipse.jdt.core.compiler.source=1.8
-~~~
-
-After you have saved the file, perform a complete project refresh in Eclipse IDE.
-
-If you are using Maven, right click your Eclipse project and select `Maven` -> `Update Project...`.
-
-You have configured everything correctly, if the following Flink program runs without exceptions:
-
-~~~java
-final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-env.fromElements(1, 2, 3).map((in) -> new Tuple1<String>(" " + in)).print();
-env.execute();
-~~~
-
-{% top %}
diff --git a/docs/dev/java_lambdas.md b/docs/dev/java_lambdas.md
new file mode 100644
index 00000000000..4b306ac455d
--- /dev/null
+++ b/docs/dev/java_lambdas.md
@@ -0,0 +1,138 @@
+---
+title: "Java Lambda Expressions"
+nav-parent_id: api-concepts
+nav-pos: 20
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Java 8 introduced several new language features designed for faster and clearer coding. With the most important feature,
+the so-called "Lambda Expressions", it opened the door to functional programming. Lambda expressions allow for implementing and
+passing functions in a straightforward way without having to declare additional (anonymous) classes.
+
+<span class="label label-danger">Attention</span> Flink supports the usage of lambda expressions for all operators of the Java API, however, whenever a lambda expression uses Java generics you need to declare type information *explicitly*.
+
+This document shows how to use lambda expressions and describes current limitations. For a general introduction to the
+Flink API, please refer to the [Programming Guide]({{ site.baseurl }}/dev/api_concepts.html)
+
+### Examples and Limitations
+
+The following example illustrates how to implement a simple, inline `map()` function that squares its input using a lambda expression.
+The types of input `i` and output parameters of the `map()` function need not to be declared as they are inferred by the Java compiler.
+
+{% highlight java %}
+env.fromElements(1, 2, 3)
+// returns the squared i
+.map(i -> i*i)
+.print();
+{% endhighlight %}
+
+Flink can automatically extract the result type information from the implementation of the method signature `OUT map(IN value)` because `OUT` is not generic but `Integer`.
+
+Unfortunately, functions such as `flatMap()` with a signature `void flatMap(IN value, Collector<OUT> out)` are compiled into `void flatMap(IN value, Collector out)` by the Java compiler. This makes it impossible for Flink to infer the type information for the output type automatically.
+
+Flink will most likely throw an exception similar to the following:
+
+{% highlight plain%}
+org.apache.flink.api.common.functions.InvalidTypesException: The generic type parameters of 'Collector' are missing.
+ In many cases lambda methods don't provide enough information for automatic type extraction when Java generics are involved.
+ An easy workaround is to use an (anonymous) class instead that implements the 'org.apache.flink.api.common.functions.FlatMapFunction' interface.
+ Otherwise the type has to be specified explicitly using type information.
+{% endhighlight %}
+
+In this case, the type information needs to be *specified explicitly*, otherwise the output will be treated as type `Object` which leads to unefficient serialization.
+
+{% highlight java %}
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.util.Collector;
+
+DataSet<Integer> input = env.fromElements(1, 2, 3);
+
+// collector type must be declared
+input.flatMap((Integer number, Collector<String> out) -> {
+ StringBuilder builder = new StringBuilder();
+ for(int i = 0; i < number; i++) {
+ builder.append("a");
+ out.collect(builder.toString());
+ }
+})
+// provide type information explicitly
+.returns(Types.STRING)
+// prints "a", "a", "aa", "a", "aa", "aaa"
+.print();
+{% endhighlight %}
+
+Similar problems occur when using a `map()` function with a generic return type. A method signature `Tuple2<Integer, Integer> map(Integer value)` is erasured to `Tuple2 map(Integer value)` in the example below.
+
+{% highlight java %}
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+env.fromElements(1, 2, 3)
+ .map(i -> Tuple2.of(i, i)) // no information about fields of Tuple2
+ .print();
+{% endhighlight %}
+
+In general, those problems can be solved in multiple ways:
+
+{% highlight java %}
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+// use the explicit ".returns(...)"
+env.fromElements(1, 2, 3)
+ .map(i -> Tuple2.of(i, i))
+ .returns(Types.TUPLE(Types.INT, Types.INT))
+ .print();
+
+// use a class instead
+env.fromElements(1, 2, 3)
+ .map(new MyTuple2Mapper())
+ .print();
+
+public static class MyTuple2Mapper extends MapFunction<Integer, Integer> {
+ @Override
+ public Tuple2<Integer, Integer> map(Integer i) {
+ return Tuple2.of(i, i);
+ }
+}
+
+// use an anonymous class instead
+env.fromElements(1, 2, 3)
+ .map(new MapFunction<Integer, Tuple2<Integer, Integer>> {
+ @Override
+ public Tuple2<Integer, Integer> map(Integer i) {
+ return Tuple2.of(i, i);
+ }
+ })
+ .print();
+
+// or in this example use a tuple subclass instead
+env.fromElements(1, 2, 3)
+ .map(i -> new DoubleTuple(i, i))
+ .print();
+
+public static class DoubleTuple extends Tuple2<Integer, Integer> {
+ public DoubleTuple(int f0, int f1) {
+ this.f0 = f0;
+ this.f1 = f1;
+ }
+}
+{% endhighlight %}
\ No newline at end of file
diff --git a/docs/dev/libs/cep.md b/docs/dev/libs/cep.md
index a6136d11e9d..6723e715e82 100644
--- a/docs/dev/libs/cep.md
+++ b/docs/dev/libs/cep.md
@@ -31,7 +31,7 @@ This page describes the API calls available in Flink CEP. We start by presenting
which allows you to specify the patterns that you want to detect in your stream, before presenting how you can
[detect and act upon matching event sequences](#detecting-patterns). We then present the assumptions the CEP
library makes when [dealing with lateness](#handling-lateness-in-event-time) in event time and how you can
-[migrate your job](#migrating-from-an-older-flink-version) from an older Flink version to Flink-1.3.
+[migrate your job](#migrating-from-an-older-flink-versionpre-13) from an older Flink version to Flink-1.3.
* This will be replaced by the TOC
{:toc}
@@ -84,7 +84,7 @@ Pattern<Event, ?> pattern = Pattern.<Event>begin("start").where(
}
}
).next("middle").subtype(SubEvent.class).where(
- new SimpleCondition<Event>() {
+ new SimpleCondition<SubEvent>() {
@Override
public boolean filter(SubEvent subEvent) {
return subEvent.getVolume() >= 10.0;
@@ -102,7 +102,7 @@ Pattern<Event, ?> pattern = Pattern.<Event>begin("start").where(
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select(
- new PatternSelectFunction<Event, Alert> {
+ new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) throws Exception {
return createAlertFrom(pattern);
@@ -115,7 +115,7 @@ DataStream<Alert> result = patternStream.select(
{% highlight scala %}
val input: DataStream[Event] = ...
-val pattern = Pattern.begin("start").where(_.getId == 42)
+val pattern = Pattern.begin[Event]("start").where(_.getId == 42)
.next("middle").subtype(classOf[SubEvent]).where(_.getVolume >= 10.0)
.followedBy("end").where(_.getName == "end")
@@ -131,7 +131,7 @@ val result: DataStream[Alert] = patternStream.select(createAlert(_))
The pattern API allows you to define complex pattern sequences that you want to extract from your input stream.
Each complex pattern sequence consists of multiple simple patterns, i.e. patterns looking for individual events with the same properties. From now on, we will call these simple patterns **patterns**, and the final complex pattern sequence we are searching for in the stream, the **pattern sequence**. You can see a pattern sequence as a graph of such patterns, where transitions from one pattern to the next occur based on user-specified
-*conditions*, e.g. `event.getName().equals("start")`. A **match** is a sequence of input events which visits all
+*conditions*, e.g. `event.getName().equals("end")`. A **match** is a sequence of input events which visits all
patterns of the complex pattern graph, through a sequence of valid pattern transitions.
{% warn Attention %} Each pattern must have a unique name, which you use later to identify the matched events.
@@ -275,36 +275,40 @@ with "foo", and if the sum of the prices of the previously accepted events for t
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-middle.oneOrMore().where(new IterativeCondition<SubEvent>() {
- @Override
- public boolean filter(SubEvent value, Context<SubEvent> ctx) throws Exception {
- if (!value.getName().startsWith("foo")) {
- return false;
- }
-
- double sum = value.getPrice();
- for (Event event : ctx.getEventsForPattern("middle")) {
- sum += event.getPrice();
+middle.oneOrMore()
+ .subtype(SubEvent.class)
+ .where(new IterativeCondition<SubEvent>() {
+ @Override
+ public boolean filter(SubEvent value, Context<SubEvent> ctx) throws Exception {
+ if (!value.getName().startsWith("foo")) {
+ return false;
+ }
+
+ double sum = value.getPrice();
+ for (Event event : ctx.getEventsForPattern("middle")) {
+ sum += event.getPrice();
+ }
+ return Double.compare(sum, 5.0) < 0;
}
- return Double.compare(sum, 5.0) < 0;
- }
-});
+ });
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-middle.oneOrMore().where(
- (value, ctx) => {
- lazy val sum = ctx.getEventsForPattern("middle").asScala.map(_.getPrice).sum
- value.getName.startsWith("foo") && sum + value.getPrice < 5.0
- }
-)
+middle.oneOrMore()
+ .subtype(classOf[SubEvent])
+ .where(
+ (value, ctx) => {
+ lazy val sum = ctx.getEventsForPattern("middle").map(_.getPrice).sum
+ value.getName.startsWith("foo") && sum + value.getPrice < 5.0
+ }
+ )
{% endhighlight %}
</div>
</div>
-{% warn Attention %} The call to `context.getEventsForPattern(...)` finds all the
+{% warn Attention %} The call to `ctx.getEventsForPattern(...)` finds all the
previously accepted events for a given potential match. The cost of this operation can vary, so when implementing
your condition, try to minimize its use.
@@ -408,7 +412,7 @@ input `"a1", "c", "a2", "b"` will have the following results:
1. **Strict Contiguity**: `{a2 b}` -- the `"c"` after `"a1"` causes `"a1"` to be discarded.
- 2. **Relaxed Contiguity**: `{a1 b}` and `{a1 a2 b}` -- `c` is ignored.
+ 2. **Relaxed Contiguity**: `{a1 b}` and `{a1 a2 b}` -- `"c"` is ignored.
3. **Non-Deterministic Relaxed Contiguity**: `{a1 b}`, `{a2 b}`, and `{a1 a2 b}`.
@@ -834,7 +838,7 @@ them between consecutive patterns, you can use:
or
1. `notNext()`, if you do not want an event type to directly follow another
-2. `notFollowedBy()`, if you do not want an event type to be anywhere between two other event types
+2. `notFollowedBy()`, if you do not want an event type to be anywhere between two other event types.
{% warn Attention %} A pattern sequence cannot end in `notFollowedBy()`.
@@ -886,14 +890,14 @@ val relaxedNot: Pattern[Event, _] = start.notFollowedBy("not").where(...)
Relaxed contiguity means that only the first succeeding matching event will be matched, while
with non-deterministic relaxed contiguity, multiple matches will be emitted for the same beginning. As an example,
-a pattern `a b`, given the event sequence `"a", "c", "b1", "b2"`, will give the following results:
+a pattern `"a b"`, given the event sequence `"a", "c", "b1", "b2"`, will give the following results:
-1. Strict Contiguity between `a` and `b`: `{}` (no match), the `"c"` after `"a"` causes `"a"` to be discarded.
+1. Strict Contiguity between `"a"` and `"b"`: `{}` (no match), the `"c"` after `"a"` causes `"a"` to be discarded.
-2. Relaxed Contiguity between `a` and `b`: `{a b1}`, as relaxed continuity is viewed as "skip non-matching events
+2. Relaxed Contiguity between `"a"` and `"b"`: `{a b1}`, as relaxed continuity is viewed as "skip non-matching events
till the next matching one".
-3. Non-Deterministic Relaxed Contiguity between `a` and `b`: `{a b1}`, `{a b2}`, as this is the most general form.
+3. Non-Deterministic Relaxed Contiguity between `"a"` and `"b"`: `{a b1}`, `{a b2}`, as this is the most general form.
It's also possible to define a temporal constraint for the pattern to be valid.
For example, you can define that a pattern should occur within 10 seconds via the `pattern.within()` method.
@@ -950,22 +954,22 @@ Pattern<Event, ?> nonDetermin = start.followedByAny(
{% highlight scala %}
val start: Pattern[Event, _] = Pattern.begin(
- Pattern.begin[Event, _]("start").where(...).followedBy("start_middle").where(...)
+ Pattern.begin[Event]("start").where(...).followedBy("start_middle").where(...)
)
// strict contiguity
val strict: Pattern[Event, _] = start.next(
- Pattern.begin[Event, _]("next_start").where(...).followedBy("next_middle").where(...)
+ Pattern.begin[Event]("next_start").where(...).followedBy("next_middle").where(...)
).times(3)
// relaxed contiguity
val relaxed: Pattern[Event, _] = start.followedBy(
- Pattern.begin[Event, _]("followedby_start").where(...).followedBy("followedby_middle").where(...)
+ Pattern.begin[Event]("followedby_start").where(...).followedBy("followedby_middle").where(...)
).oneOrMore()
// non-deterministic relaxed contiguity
val nonDetermin: Pattern[Event, _] = start.followedByAny(
- Pattern.begin[Event, _]("followedbyany_start").where(...).followedBy("followedbyany_middle").where(...)
+ Pattern.begin[Event]("followedbyany_start").where(...).followedBy("followedbyany_middle").where(...)
).optional()
{% endhighlight %}
@@ -1117,11 +1121,22 @@ pattern.within(Time.seconds(10));
</thead>
<tbody>
<tr>
- <td><strong>begin()</strong></td>
+ <td><strong>begin(#name)</strong></td>
<td>
<p>Defines a starting pattern:</p>
{% highlight scala %}
val start = Pattern.begin[Event]("start")
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>begin(#pattern_sequence)</strong></td>
+ <td>
+ <p>Defines a starting pattern:</p>
+{% highlight scala %}
+val start = Pattern.begin(
+ Pattern.begin[Event]("start").where(...).followedBy("middle").where(...)
+)
{% endhighlight %}
</td>
</tr>
@@ -1237,13 +1252,13 @@ pattern.within(Time.seconds(10))
For a given pattern, the same event may be assigned to multiple successful matches. To control to how many matches an event will be assigned, you need to specify the skip strategy called `AfterMatchSkipStrategy`. There are four types of skip strategies, listed as follows:
* <strong>*NO_SKIP*</strong>: Every possible match will be emitted.
-* <strong>*SKIP_PAST_LAST_EVENT*</strong>: Discards every partial match that contains event of the match.
-* <strong>*SKIP_TO_FIRST*</strong>: Discards every partial match that contains event of the match preceding the first of *PatternName*.
-* <strong>*SKIP_TO_LAST*</strong>: Discards every partial match that contains event of the match preceding the last of *PatternName*.
+* <strong>*SKIP_PAST_LAST_EVENT*</strong>: Discards every partial match that started after the match started but before it ended.
+* <strong>*SKIP_TO_FIRST*</strong>: Discards every partial match that started after the match started but before the first event of *PatternName* occurred.
+* <strong>*SKIP_TO_LAST*</strong>: Discards every partial match that started after the match started but before the last event of *PatternName* occurred.
Notice that when using *SKIP_TO_FIRST* and *SKIP_TO_LAST* skip strategy, a valid *PatternName* should also be specified.
-For example, for a given pattern `a b{2}` and a data stream `ab1, ab2, ab3, ab4, ab5, ab6`, the differences between these four skip strategies are as follows:
+For example, for a given pattern `b+ c` and a data stream `b1 b2 b3 c`, the differences between these four skip strategies are as follows:
<table class="table table-bordered">
<tr>
@@ -1254,38 +1269,65 @@ For example, for a given pattern `a b{2}` and a data stream `ab1, ab2, ab3, ab4,
<tr>
<td><strong>NO_SKIP</strong></td>
<td>
- <code>ab1 ab2 ab3</code><br>
- <code>ab2 ab3 ab4</code><br>
- <code>ab3 ab4 ab5</code><br>
- <code>ab4 ab5 ab6</code><br>
+ <code>b1 b2 b3 c</code><br>
+ <code>b2 b3 c</code><br>
+ <code>b3 c</code><br>
</td>
- <td>After found matching <code>ab1 ab2 ab3</code>, the match process will not discard any result.</td>
+ <td>After found matching <code>b1 b2 b3 c</code>, the match process will not discard any result.</td>
</tr>
<tr>
<td><strong>SKIP_PAST_LAST_EVENT</strong></td>
<td>
- <code>ab1 ab2 ab3</code><br>
- <code>ab4 ab5 ab6</code><br>
+ <code>b1 b2 b3 c</code><br>
</td>
- <td>After found matching <code>ab1 ab2 ab3</code>, the match process will discard all started partial matches.</td>
+ <td>After found matching <code>b1 b2 b3 c</code>, the match process will discard all started partial matches.</td>
</tr>
<tr>
- <td><strong>SKIP_TO_FIRST</strong>[<code>b</code>]</td>
+ <td><strong>SKIP_TO_FIRST</strong>[<code>b*</code>]</td>
<td>
- <code>ab1 ab2 ab3</code><br>
- <code>ab2 ab3 ab4</code><br>
- <code>ab3 ab4 ab5</code><br>
- <code>ab4 ab5 ab6</code><br>
+ <code>b1 b2 b3 c</code><br>
+ <code>b2 b3 c</code><br>
+ <code>b3 c</code><br>
</td>
- <td>After found matching <code>ab1 ab2 ab3</code>, the match process will discard all partial matches containing <code>ab1</code>, which is the only event that comes before the first <code>b</code>.</td>
+ <td>After found matching <code>b1 b2 b3 c</code>, the match process will try to discard all partial matches started before <code>b1</code>, but there are no such matches. Therefore nothing will be discarded.</td>
</tr>
<tr>
<td><strong>SKIP_TO_LAST</strong>[<code>b</code>]</td>
<td>
- <code>ab1 ab2 ab3</code><br>
- <code>ab3 ab4 ab5</code><br>
+ <code>b1 b2 b3 c</code><br>
+ <code>b3 c</code><br>
</td>
- <td>After found matching <code>ab1 ab2 ab3</code>, the match process will discard all partial matches containing <code>ab1</code> and <code>ab2</code>, which are events that comes before the last <code>b</code>.</td>
+ <td>After found matching <code>b1 b2 b3 c</code>, the match process will try to discard all partial matches started before <code>b3</code>. There is one such match <code>b2 b3 c</code></td>
+ </tr>
+</table>
+
+Have a look also at another example to better see the difference between NO_SKIP and SKIP_TO_FIRST:
+Pattern: `(a | c) (b | c) c+.greedy d` and sequence: `a b c1 c2 c3 d` Then the results will be:
+
+
+<table class="table table-bordered">
+ <tr>
+ <th class="text-left" style="width: 25%">Skip Strategy</th>
+ <th class="text-center" style="width: 25%">Result</th>
+ <th class="text-center"> Description</th>
+ </tr>
+ <tr>
+ <td><strong>NO_SKIP</strong></td>
+ <td>
+ <code>a b c1 c2 c3 d</code><br>
+ <code>b c1 c2 c3 d</code><br>
+ <code>c1 c2 c3 d</code><br>
+ <code>c2 c3 d</code><br>
+ </td>
+ <td>After found matching <code>a b c1 c2 c3 d</code>, the match process will not discard any result.</td>
+ </tr>
+ <tr>
+ <td><strong>SKIP_TO_FIRST</strong>[<code>b*</code>]</td>
+ <td>
+ <code>a b c1 c2 c3 d</code><br>
+ <code>c1 c2 c3 d</code><br>
+ </td>
+ <td>After found matching <code>a b c1 c2 c3 d</code>, the match process will try to discard all partial matches started before <code>c1</code>. There is one such match <code>b c1 c2 c3 d</code>.</td>
</tr>
</table>
@@ -1454,33 +1496,33 @@ parameters
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
OutputTag<String> outputTag = new OutputTag<String>("side-output"){};
SingleOutputStreamOperator<ComplexEvent> result = patternStream.select(
- new PatternTimeoutFunction<Event, TimeoutEvent>() {...},
outputTag,
+ new PatternTimeoutFunction<Event, TimeoutEvent>() {...},
new PatternSelectFunction<Event, ComplexEvent>() {...}
);
DataStream<TimeoutEvent> timeoutResult = result.getSideOutput(outputTag);
SingleOutputStreamOperator<ComplexEvent> flatResult = patternStream.flatSelect(
- new PatternFlatTimeoutFunction<Event, TimeoutEvent>() {...},
outputTag,
+ new PatternFlatTimeoutFunction<Event, TimeoutEvent>() {...},
new PatternFlatSelectFunction<Event, ComplexEvent>() {...}
);
DataStream<TimeoutEvent> timeoutFlatResult = flatResult.getSideOutput(outputTag);
-~~~
+{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-~~~scala
+{% highlight scala %}
val patternStream: PatternStream[Event] = CEP.pattern(input, pattern)
val outputTag = OutputTag[String]("side-output")
@@ -1492,12 +1534,12 @@ val result: SingleOutputStreamOperator[ComplexEvent] = patternStream.select(outp
}
val timeoutResult: DataStream<TimeoutEvent> = result.getSideOutput(outputTag)
-~~~
+{% endhighlight %}
The `flatSelect` API call offers the same overloaded version which takes as the first parameter a timeout function and as second parameter a selection function.
In contrast to the `select` functions, the `flatSelect` functions are called with a `Collector`. You can use the collector to emit an arbitrary number of events.
-~~~scala
+{% highlight scala %}
val patternStream: PatternStream[Event] = CEP.pattern(input, pattern)
val outputTag = OutputTag[String]("side-output")
@@ -1511,7 +1553,7 @@ val result: SingleOutputStreamOperator[ComplexEvent] = patternStream.flatSelect(
}
val timeoutResult: DataStream<TimeoutEvent> = result.getSideOutput(outputTag)
-~~~
+{% endhighlight %}
</div>
</div>
@@ -1524,7 +1566,49 @@ In `CEP` the order in which elements are processed matters. To guarantee that el
To guarantee that elements across watermarks are processed in event-time order, Flink's CEP library assumes
*correctness of the watermark*, and considers as *late* elements whose timestamp is smaller than that of the last
-seen watermark. Late elements are not further processed.
+seen watermark. Late elements are not further processed. Also, you can specify a sideOutput tag to collect the late elements come after the last seen watermark, you can use it like this.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+{% highlight java %}
+PatternStream<Event> patternStream = CEP.pattern(input, pattern);
+
+OutputTag<String> lateDataOutputTag = new OutputTag<String>("late-data"){};
+
+SingleOutputStreamOperator<ComplexEvent> result = patternStream
+ .sideOutputLateData(lateDataOutputTag)
+ .select(
+ new PatternSelectFunction<Event, ComplexEvent>() {...}
+ );
+
+DataStream<String> lateData = result.getSideOutput(lateDataOutputTag);
+
+
+{% endhighlight %}
+
+</div>
+
+<div data-lang="scala" markdown="1">
+
+{% highlight scala %}
+
+val patternStream: PatternStream[Event] = CEP.pattern(input, pattern)
+
+val lateDataOutputTag = OutputTag[String]("late-data")
+
+val result: SingleOutputStreamOperator[ComplexEvent] = patternStream
+ .sideOutputLateData(lateDataOutputTag)
+ .select{
+ pattern: Map[String, Iterable[ComplexEvent]] => ComplexEvent()
+ }
+
+val lateData: DataStream<String> = result.getSideOutput(lateDataOutputTag)
+
+{% endhighlight %}
+
+</div>
+</div>
## Examples
@@ -1580,7 +1664,7 @@ val input : DataStream[Event] = ...
val partitionedInput = input.keyBy(event => event.getId)
-val pattern = Pattern.begin("start")
+val pattern = Pattern.begin[Event]("start")
.next("middle").where(_.getName == "error")
.followedBy("end").where(_.getName == "critical")
.within(Time.seconds(10))
diff --git a/docs/dev/libs/gelly/index.md b/docs/dev/libs/gelly/index.md
index 7ae796824a0..2a08ac256d8 100644
--- a/docs/dev/libs/gelly/index.md
+++ b/docs/dev/libs/gelly/index.md
@@ -76,35 +76,35 @@ in the **opt** directory (for versions older than Flink 1.2 these can be manuall
[Maven Central](http://search.maven.org/#search|ga|1|flink%20gelly)). To run the Gelly examples the **flink-gelly** (for
Java) or **flink-gelly-scala** (for Scala) jar must be copied to Flink's **lib** directory.
-~~~bash
+{% highlight bash %}
cp opt/flink-gelly_*.jar lib/
cp opt/flink-gelly-scala_*.jar lib/
-~~~
+{% endhighlight %}
Gelly's examples jar includes drivers for each of the library methods and is provided in the **examples** directory.
After configuring and starting the cluster, list the available algorithm classes:
-~~~bash
+{% highlight bash %}
./bin/start-cluster.sh
./bin/flink run examples/gelly/flink-gelly-examples_*.jar
-~~~
+{% endhighlight %}
The Gelly drivers can generate graph data or read the edge list from a CSV file (each node in a cluster must have access
to the input file). The algorithm description, available inputs and outputs, and configuration are displayed when an
algorithm is selected. Print usage for [JaccardIndex](./library_methods.html#jaccard-index):
-~~~bash
+{% highlight bash %}
./bin/flink run examples/gelly/flink-gelly-examples_*.jar --algorithm JaccardIndex
-~~~
+{% endhighlight %}
Display [graph metrics](./library_methods.html#metric) for a million vertex graph:
-~~~bash
+{% highlight bash %}
./bin/flink run examples/gelly/flink-gelly-examples_*.jar \
--algorithm GraphMetrics --order directed \
--input RMatGraph --type integer --scale 20 --simplify directed \
--output print
-~~~
+{% endhighlight %}
The size of the graph is adjusted by the *\-\-scale* and *\-\-edge_factor* parameters. The
[library generator](./graph_generators.html#rmat-graph) provides access to additional configuration to adjust the
@@ -114,7 +114,7 @@ Sample social network data is provided by the [Stanford Network Analysis Project
The [com-lj](http://snap.stanford.edu/data/bigdata/communities/com-lj.ungraph.txt.gz) data set is a good starter size.
Run a few algorithms and monitor the job progress in Flink's Web UI:
-~~~bash
+{% highlight bash %}
wget -O - http://snap.stanford.edu/data/bigdata/communities/com-lj.ungraph.txt.gz | gunzip -c > com-lj.ungraph.txt
./bin/flink run -q examples/gelly/flink-gelly-examples_*.jar \
@@ -131,7 +131,7 @@ wget -O - http://snap.stanford.edu/data/bigdata/communities/com-lj.ungraph.txt.g
--algorithm JaccardIndex \
--input CSV --type integer --simplify undirected --input_filename com-lj.ungraph.txt --input_field_delimiter $'\t' \
--output hash
-~~~
+{% endhighlight %}
Please submit feature requests and report issues on the user [mailing list](https://flink.apache.org/community.html#mailing-lists)
or [Flink Jira](https://issues.apache.org/jira/browse/FLINK). We welcome suggestions for new algorithms and features as
diff --git a/docs/dev/libs/ml/cross_validation.md b/docs/dev/libs/ml/cross_validation.md
index e2642829833..01b01f94c6f 100644
--- a/docs/dev/libs/ml/cross_validation.md
+++ b/docs/dev/libs/ml/cross_validation.md
@@ -62,7 +62,7 @@ In a *k-fold* strategy, the DataSet is split into *k* equal subsets. Then for ea
For each training set, an algorithm is trained and then is evaluated based on the predictions based on the associated testing set. When an algorithm that has consistent grades (e.g. prediction errors) across held out datasets we can have some confidence that our approach (e.g. choice of algorithm / algorithm parameters / number of iterations) is robust against overfitting.
-<a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation">K-Fold Cross Validatation</a>
+<a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation">K-Fold Cross Validation</a>
### Multi-Random Splits
diff --git a/docs/dev/libs/ml/optimization.md b/docs/dev/libs/ml/optimization.md
index 1e3bd2ae1db..5ccde250730 100644
--- a/docs/dev/libs/ml/optimization.md
+++ b/docs/dev/libs/ml/optimization.md
@@ -225,9 +225,9 @@ The loss function which is minimized has to implement the `LossFunction` interfa
Either one defines ones own `LossFunction` or one uses the `GenericLossFunction` class which constructs the loss function from an outer loss function and a prediction function.
An example can be seen here
-```Scala
+{% highlight scala %}
val lossFunction = GenericLossFunction(SquaredLoss, LinearPrediction)
-```
+{% endhighlight %}
The full list of supported outer loss functions can be found [here](#partial-loss-function-values).
The full list of supported prediction functions can be found [here](#prediction-function-values).
diff --git a/docs/dev/libs/storm_compatibility.md b/docs/dev/libs/storm_compatibility.md
index 853b8e119ce..7d7e6c8178c 100644
--- a/docs/dev/libs/storm_compatibility.md
+++ b/docs/dev/libs/storm_compatibility.md
@@ -43,13 +43,13 @@ The code resides in the `org.apache.flink.storm` package.
Add the following dependency to your `pom.xml` if you want to execute Storm code in Flink.
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-storm{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
</dependency>
-~~~
+{% endhighlight %}
**Please note**: Do not add `storm-core` as a dependency. It is already included via `flink-storm`.
@@ -74,7 +74,7 @@ If a topology is executed in a remote cluster, parameters `nimbus.host` and `nim
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
TopologyBuilder builder = new TopologyBuilder(); // the Storm topology builder
// actual topology assembling code and used Spouts/Bolts can be used as-is
@@ -95,7 +95,7 @@ if(runLocal) { // submit to test cluster
// replaces: StormSubmitter.submitTopology(topologyId, conf, builder.createTopology());
FlinkSubmitter.submitTopology("WordCount", conf, FlinkTopology.createTopology(builder));
}
-~~~
+{% endhighlight %}
</div>
</div>
@@ -118,7 +118,7 @@ The generic type declaration `OUT` specifies the type of the source output strea
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// stream has `raw` type (single field output streams only)
@@ -128,7 +128,7 @@ DataStream<String> rawInput = env.addSource(
// process data stream
[...]
-~~~
+{% endhighlight %}
</div>
</div>
@@ -145,7 +145,7 @@ The generic type declarations `IN` and `OUT` specify the type of the operator's
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> text = env.readTextFile(localFilePath);
@@ -156,7 +156,7 @@ DataStream<Tuple2<String, Integer>> counts = text.transform(
// do further processing
[...]
-~~~
+{% endhighlight %}
</div>
</div>
@@ -192,7 +192,7 @@ Thus, Flink additionally provides `StormConfig` class that can be used like a ra
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StormConfig config = new StormConfig();
@@ -204,7 +204,7 @@ env.getConfig().setGlobalJobParameters(config);
// assemble program with embedded Spouts and/or Bolts
[...]
-~~~
+{% endhighlight %}
</div>
</div>
@@ -219,7 +219,7 @@ Furthermore, the wrapper type `SplitStreamTuple<T>` can be removed using `SplitS
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
[...]
// get DataStream from Spout or Bolt which declares two output streams s1 and s2 with output type SomeType
@@ -233,7 +233,7 @@ DataStream<SomeType> s2 = splitStream.select("s2").map(new SplitStreamMapper<Som
// do further processing on s1 and s2
[...]
-~~~
+{% endhighlight %}
</div>
</div>
@@ -259,7 +259,7 @@ An example of a finite Spout that emits records for 10 seconds only:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-~~~java
+{% highlight java %}
public class TimedFiniteSpout extends BaseRichSpout implements FiniteSpout {
[...] // implement open(), nextTuple(), ...
@@ -269,7 +269,7 @@ public class TimedFiniteSpout extends BaseRichSpout implements FiniteSpout {
return System.currentTimeMillis() - starttime > 10000l;
}
}
-~~~
+{% endhighlight %}
</div>
</div>
diff --git a/docs/dev/local_execution.md b/docs/dev/local_execution.md
index 326d5157fe3..d03029b90b4 100644
--- a/docs/dev/local_execution.md
+++ b/docs/dev/local_execution.md
@@ -43,13 +43,13 @@ Please also refer to the [debugging section]({{ site.baseurl }}/dev/batch/index.
If you are developing your program in a Maven project, you have to add the `flink-clients` module using this dependency:
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
<version>{{site.version}}</version>
</dependency>
-~~~
+{% endhighlight %}
## Local Environment
@@ -59,7 +59,7 @@ The local environment is instantiated via the method `ExecutionEnvironment.creat
In most cases, calling `ExecutionEnvironment.getExecutionEnvironment()` is the even better way to go. That method returns a `LocalEnvironment` when the program is started locally (outside the command line interface), and it returns a pre-configured environment for cluster execution, when the program is invoked by the [command line interface]({{ site.baseurl }}/ops/cli.html).
-~~~java
+{% highlight java %}
public static void main(String[] args) throws Exception {
ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment();
@@ -75,17 +75,17 @@ public static void main(String[] args) throws Exception {
JobExecutionResult res = env.execute();
}
-~~~
+{% endhighlight %}
The `JobExecutionResult` object, which is returned after the execution finished, contains the program runtime and the accumulator results.
The `LocalEnvironment` allows also to pass custom configuration values to Flink.
-~~~java
+{% highlight java %}
Configuration conf = new Configuration();
conf.setFloat(ConfigConstants.TASK_MANAGER_MEMORY_FRACTION_KEY, 0.5f);
final ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment(conf);
-~~~
+{% endhighlight %}
*Note:* The local execution environments do not start any web frontend to monitor the execution.
@@ -97,7 +97,7 @@ Users can use algorithms implemented for batch processing also for cases that ar
**Skeleton for Collection-based execution**
-~~~java
+{% highlight java %}
public static void main(String[] args) throws Exception {
// initialize a new Collection-based execution environment
final ExecutionEnvironment env = new CollectionEnvironment();
@@ -118,7 +118,7 @@ public static void main(String[] args) throws Exception {
System.err.println("Result = "+t);
}
}
-~~~
+{% endhighlight %}
The `flink-examples-batch` module contains a full example, called `CollectionExecutionExample`.
diff --git a/docs/dev/migration.md b/docs/dev/migration.md
index b9ed8fbbdd5..f91a3d8dea9 100644
--- a/docs/dev/migration.md
+++ b/docs/dev/migration.md
@@ -59,7 +59,7 @@ now clean of specific logger dependencies.
Example and quickstart archetypes already have loggers specified and should not be affected.
For other custom projects, make sure to add logger dependencies. For example, in Maven's `pom.xml`, you can add:
-~~~xml
+{% highlight xml %}
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
@@ -71,7 +71,7 @@ For other custom projects, make sure to add logger dependencies. For example, in
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
-~~~
+{% endhighlight %}
## Migrating from Flink 1.1 to Flink 1.2
diff --git a/docs/dev/restart_strategies.md b/docs/dev/restart_strategies.md
index a4901abf97b..49419e192de 100644
--- a/docs/dev/restart_strategies.md
+++ b/docs/dev/restart_strategies.md
@@ -107,9 +107,9 @@ In-between two consecutive restart attempts, the restart strategy waits a fixed
This strategy is enabled as default by setting the following configuration parameter in `flink-conf.yaml`.
-~~~
+{% highlight yaml %}
restart-strategy: fixed-delay
-~~~
+{% endhighlight %}
<table class="table table-bordered">
<thead>
@@ -135,10 +135,10 @@ restart-strategy: fixed-delay
For example:
-~~~
+{% highlight yaml %}
restart-strategy.fixed-delay.attempts: 3
restart-strategy.fixed-delay.delay: 10 s
-~~~
+{% endhighlight %}
The fixed delay restart strategy can also be set programmatically:
@@ -172,9 +172,9 @@ In-between two consecutive restart attempts, the restart strategy waits a fixed
This strategy is enabled as default by setting the following configuration parameter in `flink-conf.yaml`.
-~~~
+{% highlight yaml %}
restart-strategy: failure-rate
-~~~
+{% endhighlight %}
<table class="table table-bordered">
<thead>
@@ -203,11 +203,11 @@ restart-strategy: failure-rate
</tbody>
</table>
-~~~
+{% highlight yaml %}
restart-strategy.failure-rate.max-failures-per-interval: 3
restart-strategy.failure-rate.failure-rate-interval: 5 min
restart-strategy.failure-rate.delay: 10 s
-~~~
+{% endhighlight %}
The failure rate restart strategy can also be set programmatically:
@@ -240,9 +240,9 @@ env.setRestartStrategy(RestartStrategies.failureRateRestart(
The job fails directly and no restart is attempted.
-~~~
+{% highlight yaml %}
restart-strategy: none
-~~~
+{% endhighlight %}
The no restart strategy can also be set programmatically:
diff --git a/docs/dev/scala_shell.md b/docs/dev/scala_shell.md
index b12060b3fe4..d236430ad57 100644
--- a/docs/dev/scala_shell.md
+++ b/docs/dev/scala_shell.md
@@ -27,9 +27,9 @@ It can be used in a local setup as well as in a cluster setup.
To use the shell with an integrated Flink cluster just execute:
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh local
-~~~
+{% endhighlight %}
in the root directory of your binary Flink directory. To run the Shell on a
cluster, please see the Setup section below.
@@ -44,7 +44,7 @@ Use "benv" and "senv" to access the Batch and Streaming environment respectively
The following example will execute the wordcount program in the Scala shell:
-~~~scala
+{% highlight scala %}
Scala-Flink> val text = benv.fromElements(
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
@@ -54,21 +54,21 @@ Scala-Flink> val counts = text
.flatMap { _.toLowerCase.split("\\W+") }
.map { (_, 1) }.groupBy(0).sum(1)
Scala-Flink> counts.print()
-~~~
+{% endhighlight %}
The print() command will automatically send the specified tasks to the JobManager for execution and will show the result of the computation in the terminal.
It is possible to write results to a file. However, in this case you need to call `execute`, to run your program:
-~~~scala
+{% highlight scala %}
Scala-Flink> benv.execute("MyProgram")
-~~~
+{% endhighlight %}
### DataStream API
Similar to the batch program above, we can execute a streaming program through the DataStream API:
-~~~scala
+{% highlight scala %}
Scala-Flink> val textStreaming = senv.fromElements(
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
@@ -79,7 +79,7 @@ Scala-Flink> val countsStreaming = textStreaming
.map { (_, 1) }.keyBy(0).sum(1)
Scala-Flink> countsStreaming.print()
Scala-Flink> senv.execute("Streaming Wordcount")
-~~~
+{% endhighlight %}
Note, that in the Streaming case, the print operation does not trigger execution directly.
@@ -92,26 +92,26 @@ It is possible to add external classpaths to the Scala-shell. These will be sent
Use the parameter `-a <path/to/jar.jar>` or `--addclasspath <path/to/jar.jar>` to load additional classes.
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh [local | remote <host> <port> | yarn] --addclasspath <path/to/jar.jar>
-~~~
+{% endhighlight %}
## Setup
To get an overview of what options the Scala Shell provides, please use
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh --help
-~~~
+{% endhighlight %}
### Local
To use the shell with an integrated Flink cluster just execute:
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh local
-~~~
+{% endhighlight %}
### Remote
@@ -119,9 +119,9 @@ bin/start-scala-shell.sh local
To use it with a running cluster start the scala shell with the keyword `remote`
and supply the host and port of the JobManager with:
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh remote <hostname> <portnumber>
-~~~
+{% endhighlight %}
### Yarn Scala Shell cluster
@@ -134,9 +134,9 @@ JobManager, name of YARN application, etc.
For example, to start a Yarn cluster for the Scala Shell with two TaskManagers
use the following:
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh yarn -n 2
-~~~
+{% endhighlight %}
For all other options, see the full reference at the bottom.
@@ -146,14 +146,14 @@ For all other options, see the full reference at the bottom.
If you have previously deployed a Flink cluster using the Flink Yarn Session,
the Scala shell can connect with it using the following command:
-~~~bash
+{% highlight bash %}
bin/start-scala-shell.sh yarn
-~~~
+{% endhighlight %}
## Full Reference
-~~~bash
+{% highlight bash %}
Flink Scala Shell
Usage: start-scala-shell.sh [local|remote|yarn] [options] <args>...
@@ -175,7 +175,7 @@ Starts Flink scala shell connecting to a yarn cluster
-n arg | --container arg
Number of YARN container to allocate (= Number of TaskManagers)
-jm arg | --jobManagerMemory arg
- Memory for JobManager container [in MB]
+ Memory for JobManager container with optional unit (default: MB)
-nm <value> | --name <value>
Set a custom name for the application on YARN
-qu <arg> | --queue <arg>
@@ -183,13 +183,13 @@ Starts Flink scala shell connecting to a yarn cluster
-s <arg> | --slots <arg>
Number of slots per TaskManager
-tm <arg> | --taskManagerMemory <arg>
- Memory per TaskManager container [in MB]
+ Memory per TaskManager container with optional unit (default: MB)
-a <path/to/jar> | --addclasspath <path/to/jar>
Specifies additional jars to be used in Flink
--configDir <value>
The configuration directory.
-h | --help
Prints this usage text
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/dev/stream/operators/asyncio.md b/docs/dev/stream/operators/asyncio.md
index 64570740531..e92e7a96d34 100644
--- a/docs/dev/stream/operators/asyncio.md
+++ b/docs/dev/stream/operators/asyncio.md
@@ -154,11 +154,11 @@ class AsyncDatabaseRequest extends AsyncFunction[String, (String, String)] {
override def asyncInvoke(str: String, resultFuture: ResultFuture[(String, String)]): Unit = {
// issue the asynchronous request, receive a future for the result
- val resultFuture: Future[String] = client.query(str)
+ val resultFutureRequested: Future[String] = client.query(str)
// set the callback to be executed once the request by the client is complete
// the callback simply forwards the result to the result future
- resultFuture.onSuccess {
+ resultFutureRequested.onSuccess {
case result: String => resultFuture.complete(Iterable((str, result)))
}
}
@@ -190,6 +190,12 @@ The following two parameters control the asynchronous operations:
is exhausted.
+### Timeout Handling
+
+When an async I/O request times out, by default an exception is thrown and job is restarted.
+If you want to handle timeouts, you can override the `AsyncFunction#timeout` method.
+
+
### Order of Results
The concurrent requests issued by the `AsyncFunction` frequently complete in some undefined order, based on which request finished first.
diff --git a/docs/dev/stream/operators/index.md b/docs/dev/stream/operators/index.md
index 1dbdef421ba..422dbbf9136 100644
--- a/docs/dev/stream/operators/index.md
+++ b/docs/dev/stream/operators/index.md
@@ -306,6 +306,21 @@ dataStream.join(otherStream)
.where(<key selector>).equalTo(<key selector>)
.window(TumblingEventTimeWindows.of(Time.seconds(3)))
.apply (new JoinFunction () {...});
+ {% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>Interval Join</strong><br>KeyedStream,KeyedStream → DataStream</td>
+ <td>
+ <p>Join two elements e1 and e2 of two keyed streams with a common key over a given time interval, so that e1.timestamp + lowerBound <= e2.timestamp <= e1.timestamp + upperBound</p>
+ {% highlight java %}
+// this will join the two streams so that
+// key1 == key2 && leftTs - 2 < rightTs < leftTs + 2
+keyedStream.intervalJoin(otherKeyedStream)
+ .between(Time.milliseconds(-2), Time.milliseconds(2)) // lower and upper bound
+ .upperBoundExclusive(true) // optional
+ .lowerBoundExclusive(true) // optional
+ .process(new IntervalJoinFunction() {...});
{% endhighlight %}
</td>
</tr>
diff --git a/docs/dev/stream/operators/process_function.md b/docs/dev/stream/operators/process_function.md
index 1ed4edf7c94..4f36721adc7 100644
--- a/docs/dev/stream/operators/process_function.md
+++ b/docs/dev/stream/operators/process_function.md
@@ -271,22 +271,33 @@ override def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[OUT]):
</div>
</div>
-## Optimizations
+## Timers
+
+Both types of timers (processing-time and event-time) are internally maintained by the `TimerService` and enqueued for execution.
+
+The `TimerService` deduplicates timers per key and timestamp, i.e., there is at most one timer per key and timestamp. If multiple timers are registered for the same timestamp, the `onTimer()` method will be called just once.
+
+**Note:** Flink synchronizes invocations of `onTimer()` and `processElement()`. Hence, users do not have to worry about concurrent modification of state.
+
+### Fault Tolerance
+
+Timers are fault tolerant and checkpointed along with the state of the application.
+In case of a failure recovery or when starting an application from a savepoint, the timers are restored.
+
+**Note:** Checkpointed processing-time timers that were supposed to fire before their restoration, will fire immediately.
+This might happen when an application recovers from a failure or when it is started from a savepoint.
+
+**Note:** Timers are always synchronously checkpointed, regardless of the configuration of the state backends.
+Therefore, a large number of timers can significantly increase checkpointing time.
+See the "Timer Coalescing" section for advice on how to reduce the number of timers.
### Timer Coalescing
-Every timer registered at the `TimerService` via `registerEventTimeTimer()` or
-`registerProcessingTimeTimer()` will be stored on the Java heap and enqueued for execution. There is,
-however, a maximum of one timer per key and timestamp at a millisecond resolution and thus, in the
-worst case, every key may have a timer for each upcoming millisecond. Even if you do not do any
-processing for outdated timers in `onTimer`, this may put a significant burden on the
-Flink runtime.
-
-Since there is only one timer per key and timestamp, however, you may coalesce timers by reducing the
-timer resolution. For a timer resolution of 1 second (event or processing time), for example, you
-can round down the target time to full seconds and therefore allow the timer to fire at most 1
-second earlier but not later than with millisecond accuracy. As a result, there would be at most
-one timer for each combination of key and timestamp:
+Since Flink maintains only one timer per key and timestamp, you can reduce the number of timers by reducing the timer resolution to coalesce them.
+
+For a timer resolution of 1 second (event or processing time), you
+can round down the target time to full seconds. Timers will fire at most 1 second earlier but not later than requested with millisecond accuracy.
+As a result, there are at most one timer per key and second.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
diff --git a/docs/dev/stream/operators/windows.md b/docs/dev/stream/operators/windows.md
index 649bfe861c9..f657c80379d 100644
--- a/docs/dev/stream/operators/windows.md
+++ b/docs/dev/stream/operators/windows.md
@@ -504,7 +504,7 @@ private static class AverageAggregate
@Override
public Double getResult(Tuple2<Long, Long> accumulator) {
- return accumulator.f0 / accumulator.f1;
+ return ((double) accumulator.f0) / accumulator.f1;
}
@Override
@@ -730,7 +730,7 @@ input
/* ... */
-public class MyProcessWindowFunction implements ProcessWindowFunction<Tuple<String, Long>, String, String, TimeWindow> {
+public class MyProcessWindowFunction extends ProcessWindowFunction<Tuple<String, Long>, String, String, TimeWindow> {
void process(String key, Context context, Iterable<Tuple<String, Long>> input, Collector<String> out) {
long count = 0;
@@ -778,7 +778,7 @@ The example shows a `ProcessWindowFunction` that counts the elements in a window
A `ProcessWindowFunction` can be combined with either a `ReduceFunction`, an `AggregateFunction`, or a `FoldFunction` to
incrementally aggregate elements as they arrive in the window.
When the window is closed, the `ProcessWindowFunction` will be provided with the aggregated result.
-This allows to incrementally compute windows while having access to the
+This allows it to incrementally compute windows while having access to the
additional window meta information of the `ProcessWindowFunction`.
<span class="label label-info">Note</span> You can also the legacy `WindowFunction` instead of
@@ -797,7 +797,7 @@ DataStream<SensorReading> input = ...;
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.reduce(new MyReduceFunction(), new MyProcessWindowFunction());
// Function definitions
@@ -810,7 +810,7 @@ private static class MyReduceFunction implements ReduceFunction<SensorReading> {
}
private static class MyProcessWindowFunction
- implements ProcessWindowFunction<SensorReading, Tuple2<Long, SensorReading>, String, TimeWindow> {
+ extends ProcessWindowFunction<SensorReading, Tuple2<Long, SensorReading>, String, TimeWindow> {
public void process(String key,
Context context,
@@ -830,7 +830,7 @@ val input: DataStream[SensorReading] = ...
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.reduce(
(r1: SensorReading, r2: SensorReading) => { if (r1.value > r2.value) r2 else r1 },
( key: String,
@@ -856,11 +856,11 @@ the average.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-DataStream<Tuple2<String, Long> input = ...;
+DataStream<Tuple2<String, Long>> input = ...;
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.aggregate(new AverageAggregate(), new MyProcessWindowFunction());
// Function definitions
@@ -883,7 +883,7 @@ private static class AverageAggregate
@Override
public Double getResult(Tuple2<Long, Long> accumulator) {
- return accumulator.f0 / accumulator.f1;
+ return ((double) accumulator.f0) / accumulator.f1;
}
@Override
@@ -893,7 +893,7 @@ private static class AverageAggregate
}
private static class MyProcessWindowFunction
- implements ProcessWindowFunction<Double, Tuple2<String, Double>, String, TimeWindow> {
+ extends ProcessWindowFunction<Double, Tuple2<String, Double>, String, TimeWindow> {
public void process(String key,
Context context,
@@ -913,7 +913,7 @@ val input: DataStream[(String, Long)] = ...
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.aggregate(new AverageAggregate(), new MyProcessWindowFunction())
// Function definitions
@@ -959,7 +959,7 @@ DataStream<SensorReading> input = ...;
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.fold(new Tuple3<String, Long, Integer>("",0L, 0), new MyFoldFunction(), new MyProcessWindowFunction())
// Function definitions
@@ -975,7 +975,7 @@ private static class MyFoldFunction
}
private static class MyProcessWindowFunction
- implements ProcessWindowFunction<Tuple3<String, Long, Integer>, Tuple3<String, Long, Integer>, String, TimeWindow> {
+ extends ProcessWindowFunction<Tuple3<String, Long, Integer>, Tuple3<String, Long, Integer>, String, TimeWindow> {
public void process(String key,
Context context,
@@ -995,7 +995,7 @@ val input: DataStream[SensorReading] = ...
input
.keyBy(<key selector>)
- .timeWindow(<window assigner>)
+ .timeWindow(<duration>)
.fold (
("", 0L, 0),
(acc: (String, Long, Int), r: SensorReading) => { ("", 0L, acc._3 + 1) },
@@ -1412,7 +1412,6 @@ val globalResults = resultsPerKey
In this example, the results for time window `[0, 5)` from the first operation will also end up in
time window `[0, 5)` in the subsequent windowed operation. This allows calculating a sum per key
and then calculating the top-k elements within the same window in the second operation.
-and then calculating the top-k elements within the same window in the second operation.
## Useful state size considerations
diff --git a/docs/dev/stream/state/broadcast_state.md b/docs/dev/stream/state/broadcast_state.md
new file mode 100644
index 00000000000..593b6a29a87
--- /dev/null
+++ b/docs/dev/stream/state/broadcast_state.md
@@ -0,0 +1,279 @@
+---
+title: "The Broadcast State Pattern"
+nav-parent_id: streaming_state
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+[Working with State](state.html) describes operator state which upon restore is either evenly distributed among the
+parallel tasks of an operator, or unioned, with the whole state being used to initialize the restored parallel tasks.
+
+A third type of supported *operator state* is the *Broadcast State*. Broadcast state was introduced to support use cases
+where some data coming from one stream is required to be broadcasted to all downstream tasks, where it is stored locally
+and is used to process all incoming elements on the other stream. As an example where broadcast state can emerge as a
+natural fit, one can imagine a low-throughput stream containing a set of rules which we want to evaluate against all
+elements coming from another stream. Having the above type of use cases in mind, broadcast state differs from the rest
+of operator states in that:
+ 1. it has a map format,
+ 2. it is only available to specific operators that have as inputs a *broadcasted* stream and a *non-broadcasted* one, and
+ 3. such an operator can have *multiple broadcast states* with different names.
+
+## Provided APIs
+
+To show the provided APIs, we will start with an example before presenting their full functionality. As our running
+example, we will use the case where we have a stream of objects of different colors and shapes and we want to find pairs
+of objects of the same color that follow a certain pattern, *e.g.* a rectangle followed by a triangle. We assume that
+the set of interesting patterns evolves over time.
+
+In this example, the first stream will contain elements of type `Item` with a `Color` and a `Shape` property. The other
+stream will contain the `Rules`.
+
+Starting from the stream of `Items`, we just need to *key it* by `Color`, as we want pairs of the same color. This will
+make sure that elements of the same color end up on the same physical machine.
+
+{% highlight java %}
+// key the shapes by color
+KeyedStream<Item, Color> colorPartitionedStream = shapeStream
+ .keyBy(new KeySelector<Shape, Color>(){...});
+{% endhighlight %}
+
+Moving on to the `Rules`, the stream containing them should be broadcasted to all downstream tasks, and these tasks
+should store them locally so that they can evaluate them against all incoming `Items`. The snippet below will i) broadcast
+the stream of rules and ii) using the provided `MapStateDescriptor`, it will create the broadcast state where the rules
+will be stored.
+
+{% highlight java %}
+
+// a map descriptor to store the name of the rule (string) and the rule itself.
+MapStateDescriptor<String, Rule> ruleStateDescriptor = new MapStateDescriptor<>(
+ "RulesBroadcastState",
+ BasicTypeInfo.STRING_TYPE_INFO,
+ TypeInformation.of(new TypeHint<Rule>() {}));
+
+// broadcast the rules and create the broadcast state
+BroadcastStream<Rule> ruleBroadcastStream = ruleStream
+ .broadcast(ruleStateDescriptor);
+{% endhighlight %}
+
+Finally, in order to evaluate the `Rules` against the incoming elements from the `Item` stream, we need to:
+ 1) connect the two streams and
+ 2) specify our match detecting logic.
+
+Connecting a stream (keyed or non-keyed) with a `BroadcastStream` can be done by calling `connect()` on the
+non-broadcasted stream, with the `BroadcastStream` as an argument. This will return a `BroadcastConnectedStream`, on
+which we can call `process()` with a special type of `CoProcessFunction`. The function will contain our matching logic.
+The exact type of the function depends on the type of the non-broadcasted stream:
+ - if that is **keyed**, then the function is a `KeyedBroadcastProcessFunction`.
+ - if it is **non-keyed**, the function is a `BroadcastProcessFunction`.
+
+ Given that our non-broadcasted stream is keyed, the following snippet includes the above calls:
+
+<div class="alert alert-info">
+ <strong>Attention:</strong> The connect should be called on the non-broadcasted stream, with the BroadcastStream
+ as an argument.
+</div>
+
+{% highlight java %}
+DataStream<Match> output = colorPartitionedStream
+ .connect(ruleBroadcastStream)
+ .process(
+
+ // type arguments in our KeyedBroadcastProcessFunction represent:
+ // 1. the key of the keyed stream
+ // 2. the type of elements in the non-broadcast side
+ // 3. the type of elements in the broadcast side
+ // 4. the type of the result, here a string
+
+ new KeyedBroadcastProcessFunction<Color, Item, Rule, String>() {
+ // my matching logic
+ }
+ )
+{% endhighlight %}
+
+### BroadcastProcessFunction and KeyedBroadcastProcessFunction
+
+As in the case of a `CoProcessFunction`, these functions have two process methods to implement; the `processBroadcastElement()`
+which is responsible for processing incoming elements in the broadcasted stream and the `processElement()` which is used
+for the non-broadcasted one. The full signatures of the methods are presented below:
+
+{% highlight java %}
+public abstract class BroadcastProcessFunction<IN1, IN2, OUT> extends BaseBroadcastProcessFunction {
+
+ public abstract void processElement(IN1 value, ReadOnlyContext ctx, Collector<OUT> out) throws Exception;
+
+ public abstract void processBroadcastElement(IN2 value, Context ctx, Collector<OUT> out) throws Exception;
+}
+{% endhighlight %}
+
+{% highlight java %}
+public abstract class KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> {
+
+ public abstract void processElement(IN1 value, ReadOnlyContext ctx, Collector<OUT> out) throws Exception;
+
+ public abstract void processBroadcastElement(IN2 value, Context ctx, Collector<OUT> out) throws Exception;
+
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<OUT> out) throws Exception;
+}
+{% endhighlight %}
+
+The first thing to notice is that both functions require the implementation of the `processBroadcastElement()` method
+for processing elements in the broadcast side and the `processElement()` for elements in the non-broadcasted side.
+
+The two methods differ in the context they are provided. The non-broadcast side has a `ReadOnlyContext`, while the
+broadcasted side has a `Context`.
+
+Both of these contexts (`ctx` in the following enumeration):
+ 1. give access to the broadcast state: `ctx.getBroadcastState(MapStateDescriptor<K, V> stateDescriptor)`
+ 2. allow to query the timestamp of the element: `ctx.timestamp()`,
+ 3. get the current watermark: `ctx.currentWatermark()`
+ 4. get the current processing time: `ctx.currentProcessingTime()`, and
+ 5. emit elements to side-outputs: `ctx.output(OutputTag<X> outputTag, X value)`.
+
+The `stateDescriptor` in the `getBroadcastState()` should be identical to the one in the `.broadcast(ruleStateDescriptor)`
+above.
+
+The difference lies in the type of access each one gives to the broadcast state. The broadcasted side has
+**read-write access** to it, while the non-broadcast side has **read-only access** (thus the names). The reason for this
+is that in Flink there is no cross-task communication. So, to guarantee that the contents in the Broadcast State are the
+same across all parallel instances of our operator, we give read-write access only to the broadcast side, which sees the
+same elements across all tasks, and we require the computation on each incoming element on that side to be identical
+across all tasks. Ignoring this rule would break the consistency guarantees of the state, leading to inconsistent and
+often difficult to debug results.
+
+<div class="alert alert-info">
+ <strong>Attention:</strong> The logic implemented in `processBroadcast()` must have the same determinstic behavior
+ across all parallel instances!
+</div>
+
+Finally, due to the fact that the `KeyedBroadcastProcessFunction` is operating on a keyed stream, it
+exposes some functionality which is not available to the `BroadcastProcessFunction`. That is:
+ 1. the `ReadOnlyContext` in the `processElement()` method gives access to Flink's underlying timer service, which allows
+ to register event and/or processing time timers. When a timer fires, the `onTimer()` (shown above) is invoked with an
+ `OnTimerContext` which exposes the same functionality as the `ReadOnlyContext` plus
+ - the ability to ask if the timer that fired was an event or processing time one and
+ - to query the key associated with the timer.
+
+ This is aligned with the `onTimer()` method of the `KeyedProcessFunction`.
+ 2. the `Context` in the `processBroadcastElement()` method contains the method
+ `applyToKeyedState(StateDescriptor<S, VS> stateDescriptor, KeyedStateFunction<KS, S> function)`. This allows to
+ register a `KeyedStateFunction` to be **applied to all states of all keys** associated with the provided `stateDescriptor`.
+
+<div class="alert alert-info">
+ <strong>Attention:</strong> Registering timers is only possible at `processElement()` of the `KeyedBroadcastProcessFunction`
+ and only there. It is not possible in the `processBroadcastElement()` method, as there is no key associated to the
+ broadcasted elements.
+</div>
+
+Coming back to our original example, our `KeyedBroadcastProcessFunction` could look like the following:
+
+{% highlight java %}
+new KeyedBroadcastProcessFunction<Color, Item, Rule, String>() {
+
+ // store partial matches, i.e. first elements of the pair waiting for their second element
+ // we keep a list as we may have many first elements waiting
+ private final MapStateDescriptor<String, List<Item>> mapStateDesc =
+ new MapStateDescriptor<>(
+ "items",
+ BasicTypeInfo.STRING_TYPE_INFO,
+ new ListTypeInfo<>(Item.class));
+
+ // identical to our ruleStateDescriptor above
+ private final MapStateDescriptor<String, Rule> ruleStateDescriptor =
+ new MapStateDescriptor<>(
+ "RulesBroadcastState",
+ BasicTypeInfo.STRING_TYPE_INFO,
+ TypeInformation.of(new TypeHint<Rule>() {}));
+
+ @Override
+ public void processBroadcastElement(Rule value,
+ Context ctx,
+ Collector<String> out) throws Exception {
+ ctx.getBroadcastState(ruleStateDescriptor).put(value.name, value);
+ }
+
+ @Override
+ public void processElement(Item value,
+ ReadOnlyContext ctx,
+ Collector<String> out) throws Exception {
+
+ final MapState<String, List<Item>> state = getRuntimeContext().getMapState(mapStateDesc);
+ final Shape shape = value.getShape();
+
+ for (Map.Entry<String, Rule> entry:
+ ctx.getBroadcastState(ruleStateDescriptor).immutableEntries()) {
+ final String ruleName = entry.getKey();
+ final Rule rule = entry.getValue();
+
+ List<Item> stored = state.get(ruleName);
+ if (stored == null) {
+ stored = new ArrayList<>();
+ }
+
+ if (shape == rule.second && !stored.isEmpty()) {
+ for (Item i : stored) {
+ out.collect("MATCH: " + i + " - " + value);
+ }
+ stored.clear();
+ }
+
+ // there is no else{} to cover if rule.first == rule.second
+ if (shape.equals(rule.first)) {
+ stored.add(value);
+ }
+
+ if (stored.isEmpty()) {
+ state.remove(ruleName);
+ } else {
+ state.put(ruleName, stored);
+ }
+ }
+ }
+}
+{% endhighlight %}
+
+## Important Considerations
+
+After describing the offered APIs, this section focuses on the important things to keep in mind when using broadcast
+state. These are:
+
+ - **There is no cross-task communication:** As stated earlier, this is the reason why only the broadcast side of a
+`(Keyed)-BroadcastProcessFunction` can modify the contents of the broadcast state. In addition, the user has to make
+sure that all tasks modify the contents of the broadcast state in the same way for each incoming element. Otherwise,
+different tasks might have different contents, leading to inconsistent results.
+
+ - **Order of events in Broadcast State may differ across tasks:** Although broadcasting the elements of a stream
+guarantees that all elements will (eventually) go to all downstream tasks, elements may arrive in a different order
+to each task. So the state updates for each incoming element *MUST NOT depend on the ordering* of the incoming
+events.
+
+ - **All tasks checkpoint their broadcast state:** Although all tasks have the same elements in their broadcast state
+when a checkpoint takes place (checkpoint barriers do not overpass elements), all tasks checkpoint their broadcast state,
+and not just one of them. This is a design decision to avoid having all tasks read from the same file during a restore
+(thus avoiding hotspots), although it comes at the expense of increasing the size of the checkpointed state by a factor
+of p (= parallelism). Flink guarantees that upon restoring/rescaling there will be **no duplicates** and **no missing data**.
+In case of recovery with the same or smaller parallelism, each task reads its checkpointed state. Upon scaling up, each
+task reads its own state, and the remaining tasks (`p_new`-`p_old`) read checkpoints of previous tasks in a round-robin
+manner.
+
+ - **No RocksDB state backend:** Broadcast state is kept in-memory at runtime and memory provisioning should be done
+accordingly. This holds for all operator states.
diff --git a/docs/dev/stream/state/checkpointing.md b/docs/dev/stream/state/checkpointing.md
index f856d7d2351..c8c17f1f2b9 100644
--- a/docs/dev/stream/state/checkpointing.md
+++ b/docs/dev/stream/state/checkpointing.md
@@ -1,7 +1,7 @@
---
title: "Checkpointing"
nav-parent_id: streaming_state
-nav-pos: 2
+nav-pos: 3
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -133,17 +133,7 @@ env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see [configuration]({{ site.baseurl }}/ops/config.html) for a full guide):
-- `state.backend`: The backend that will be used to store operator state checkpoints if checkpointing is enabled. Supported backends:
- - `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
- - `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
-
-- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
-
-- `state.backend.rocksdb.checkpointdir`: The local directory for storing RocksDB files, or a list of directories separated by the systems directory delimiter (for example ‘:’ (colon) on Linux/Unix). (DEFAULT value is `taskmanager.tmp.dirs`)
-
-- `state.checkpoints.dir`: The target directory for meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints).
-
-- `state.checkpoints.num-retained`: The number of completed checkpoint instances to retain. Having more than one allows recovery fallback to an earlier checkpoints if the latest checkpoint is corrupt. (Default: 1)
+{% include generated/checkpointing_configuration.html %}
{% top %}
diff --git a/docs/dev/stream/state/custom_serialization.md b/docs/dev/stream/state/custom_serialization.md
index 7f886d2025f..f3941bc0c98 100644
--- a/docs/dev/stream/state/custom_serialization.md
+++ b/docs/dev/stream/state/custom_serialization.md
@@ -2,7 +2,7 @@
title: "Custom Serialization for Managed State"
nav-title: "Custom Serialization"
nav-parent_id: streaming_state
-nav-pos: 10
+nav-pos: 6
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
diff --git a/docs/dev/stream/state/index.md b/docs/dev/stream/state/index.md
index fe725afc210..f9a24c3d039 100644
--- a/docs/dev/stream/state/index.md
+++ b/docs/dev/stream/state/index.md
@@ -49,6 +49,7 @@ Where to go next?
-----------------
* [Working with State](state.html): Shows how to use state in a Flink application and explains the different kinds of state.
+* [The Broadcast State Pattern](broadcast_state.html): Explains how to connect a broadcast stream with a non-broadcast stream and use state to exchange information between them.
* [Checkpointing](checkpointing.html): Describes how to enable and configure checkpointing for fault tolerance.
* [Queryable State](queryable_state.html): Explains how to access state from outside of Flink during runtime.
* [Custom Serialization for Managed State](custom_serialization.html): Discusses custom serialization logic for state and its upgrades.
diff --git a/docs/dev/stream/state/queryable_state.md b/docs/dev/stream/state/queryable_state.md
index af646dfa7a4..4fd2186626d 100644
--- a/docs/dev/stream/state/queryable_state.md
+++ b/docs/dev/stream/state/queryable_state.md
@@ -1,7 +1,7 @@
---
title: "Queryable State"
nav-parent_id: streaming_state
-nav-pos: 3
+nav-pos: 4
is_beta: true
---
<!--
@@ -60,7 +60,7 @@ The Queryable State feature consists of three main entities:
returning it to the client, and
3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
-In a nutshell, the client will connect to one of the proxies and send a request for the state associated with a specific
+The client connects to one of the proxies and sends a request for the state associated with a specific
key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in
*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
@@ -143,8 +143,7 @@ can be made queryable by making the appropriate state descriptor queryable via
ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
new ValueStateDescriptor<>(
"average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}), // type information
- Tuple2.of(0L, 0L)); // default value of the state, if nothing was set
+ TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
descriptor.setQueryable("query-name"); // queryable state name
{% endhighlight %}
@@ -162,14 +161,19 @@ So far, you have set up your cluster to run with queryable state and you have de
queryable. Now it is time to see how to query this state.
For this you can use the `QueryableStateClient` helper class. This is available in the `flink-queryable-state-client`
-jar which you have to explicitly include as a dependency in the `pom.xml` of your project, as shown below:
+jar which must be explicitly included as a dependency in the `pom.xml` of your project along with `flink-core`, as shown below:
<div data-lang="java" markdown="1">
{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-queryable-state-client-java_{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
+ <artifactId>flink-core</artifactId>
+ <version>{{ site.version }}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-queryable-state-client-java{{ site.scala_version_suffix }}</artifactId>
+ <version>{{ site.version }}</version>
</dependency>
{% endhighlight %}
</div>
@@ -189,11 +193,11 @@ With the client ready, to query a state of type `V`, associated with a key of ty
{% highlight java %}
CompletableFuture<S> getKvState(
- final JobID jobId,
- final String queryableStateName,
- final K key,
- final TypeInformation<K> keyTypeInfo,
- final StateDescriptor<S, V> stateDescriptor)
+ JobID jobId,
+ String queryableStateName,
+ K key,
+ TypeInformation<K> keyTypeInfo,
+ StateDescriptor<S, V> stateDescriptor)
{% endhighlight %}
The above returns a `CompletableFuture` eventually holding the state value for the queryable state instance identified
@@ -207,7 +211,7 @@ the actual value. This can be any of the state types supported by Flink: `ValueS
`AggregatingState`, and the currently deprecated `FoldingState`.
<div class="alert alert-info">
- <strong>Note:</strong>These state objects do not allow modifications to the contained state. You can use them to get
+ <strong>Note:</strong> These state objects do not allow modifications to the contained state. You can use them to get
the actual value of the state, <i>e.g.</i> using <code>valueState.get()</code>, or iterate over
the contained <code><K, V></code> entries, <i>e.g.</i> using the <code>mapState.entries()</code>, but you cannot
modify them. As an example, calling the <code>add()</code> method on a returned list state will throw an
@@ -224,7 +228,7 @@ the actual value. This can be any of the state types supported by Flink: `ValueS
The following example extends the `CountWindowAverage` example
(see [Using Managed Keyed State]({{ site.baseurl }}/dev/stream/state/state.html#using-managed-keyed-state))
-by making it queryable and showing how to query this value:
+by making it queryable and shows how to query this value:
{% highlight java %}
public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>> {
@@ -249,8 +253,7 @@ public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>,
ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
new ValueStateDescriptor<>(
"average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}), // type information
- Tuple2.of(0L, 0L)); // default value of the state, if nothing was set
+ TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
descriptor.setQueryable("query-name");
sum = getRuntimeContext().getState(descriptor);
}
@@ -266,8 +269,7 @@ QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort);
ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
new ValueStateDescriptor<>(
"average",
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}),
- Tuple2.of(0L, 0L));
+ TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}));
CompletableFuture<ValueState<Tuple2<Long, Long>>> resultFuture =
client.getKvState(jobId, "query-name", key, BasicTypeInfo.LONG_TYPE_INFO, descriptor);
diff --git a/docs/dev/stream/state/state.md b/docs/dev/stream/state/state.md
index a26e24d8702..44a3653a61f 100644
--- a/docs/dev/stream/state/state.md
+++ b/docs/dev/stream/state/state.md
@@ -368,7 +368,7 @@ public class BufferingSink
ListStateDescriptor<Tuple2<String, Integer>> descriptor =
new ListStateDescriptor<>(
"buffered-elements",
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}));
+ TypeInformation.of(new TypeHint<Tuple2<String, Integer>>() {}));
checkpointedState = context.getOperatorStateStore().getListState(descriptor);
diff --git a/docs/dev/stream/testing.md b/docs/dev/stream/testing.md
index 0a43d313e6d..7676b199c22 100644
--- a/docs/dev/stream/testing.md
+++ b/docs/dev/stream/testing.md
@@ -140,7 +140,7 @@ You could write the following integration test:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-public class ExampleIntegrationTest extends StreamingMultipleProgramsTestBase {
+public class ExampleIntegrationTest extends AbstractTestBase {
@Test
public void testMultiply() throws Exception {
@@ -181,7 +181,7 @@ public class ExampleIntegrationTest extends StreamingMultipleProgramsTestBase {
<div data-lang="scala" markdown="1">
{% highlight scala %}
-class ExampleIntegrationTest extends StreamingMultipleProgramsTestBase {
+class ExampleIntegrationTest extends AbstractTestBase {
@Test
def testMultiply(): Unit = {
diff --git a/docs/dev/table/common.md b/docs/dev/table/common.md
index 3e3965a0d46..146d1a6c1fd 100644
--- a/docs/dev/table/common.md
+++ b/docs/dev/table/common.md
@@ -173,7 +173,7 @@ A `Table` is registered in a `TableEnvironment` as follows:
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// Table is the result of a simple projection query
-Table projTable = tableEnv.scan("X").project(...);
+Table projTable = tableEnv.scan("X").select(...);
// register the Table projTable as table "projectedX"
tableEnv.registerTable("projectedTable", projTable);
@@ -186,7 +186,7 @@ tableEnv.registerTable("projectedTable", projTable);
val tableEnv = TableEnvironment.getTableEnvironment(env)
// Table is the result of a simple projection query
-val projTable: Table = tableEnv.scan("X").project(...)
+val projTable: Table = tableEnv.scan("X").select(...)
// register the Table projTable as table "projectedX"
tableEnv.registerTable("projectedTable", projTable)
@@ -365,9 +365,9 @@ val tableEnv = TableEnvironment.getTableEnvironment(env)
// register Orders table
// scan registered Orders table
-Table orders = tableEnv.scan("Orders")
+val orders = tableEnv.scan("Orders")
// compute revenue for all customers from France
-Table revenue = orders
+val revenue = orders
.filter('cCountry === "FRANCE")
.groupBy('cID, 'cName)
.select('cID, 'cName, 'revenue.sum AS 'revSum)
@@ -419,7 +419,7 @@ val tableEnv = TableEnvironment.getTableEnvironment(env)
// register Orders table
// compute revenue for all customers from France
-Table revenue = tableEnv.sqlQuery("""
+val revenue = tableEnv.sqlQuery("""
|SELECT cID, cName, SUM(revenue) AS revSum
|FROM Orders
|WHERE cCountry = 'FRANCE'
diff --git a/docs/dev/table/sourceSinks.md b/docs/dev/table/sourceSinks.md
index 792f19d260a..4c5f2e27234 100644
--- a/docs/dev/table/sourceSinks.md
+++ b/docs/dev/table/sourceSinks.md
@@ -460,7 +460,8 @@ val source: KafkaTableSource = Kafka010JsonTableSource.builder()
Flink provides `TimestampExtractor` implementations for common use cases.
The following `TimestampExtractor` implementations are currently available:
-* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG` or `SQL_TIMESTAMP` field.
+* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG` or `SQL_TIMESTAMP`, or ISO date formatted `STRING` field.
+ * One example of ISO date format would be '2018-05-28 12:34:56.000'.
* `StreamRecordTimestamp()`: Extracts the value of a rowtime attribute from the timestamp of the `DataStream` `StreamRecord`. Note, this `TimestampExtractor` is not available for batch table sources.
A custom `TimestampExtractor` can be defined by implementing the corresponding interface.
diff --git a/docs/dev/table/sql.md b/docs/dev/table/sql.md
index 27d79af0dc3..5b83e9d23cb 100644
--- a/docs/dev/table/sql.md
+++ b/docs/dev/table/sql.md
@@ -24,7 +24,7 @@ under the License.
SQL queries are specified with the `sqlQuery()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in [subsequent SQL and Table API queries](common.html#mixing-table-api-and-sql), be [converted into a DataSet or DataStream](common.html#integration-with-datastream-and-dataset-api), or [written to a TableSink](common.html#emit-a-table)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.
-In order to access a table in a SQL query, it must be [registered in the TableEnvironment](common.html#register-a-table-in-the-catalog). A table can be registered from a [TableSource](common.html#register-a-tablesource), [Table](common.html#register-a-table), [DataStream, or DataSet](common.html#register-a-datastream-or-dataset-as-table). Alternatively, users can also [register external catalogs in a TableEnvironment](common.html#register-an-external-catalog) to specify the location of the data sources.
+In order to access a table in a SQL query, it must be [registered in the TableEnvironment](common.html#register-tables-in-the-catalog). A table can be registered from a [TableSource](common.html#register-a-tablesource), [Table](common.html#register-a-table), [DataStream, or DataSet](common.html#register-a-datastream-or-dataset-as-table). Alternatively, users can also [register external catalogs in a TableEnvironment](common.html#register-an-external-catalog) to specify the location of the data sources.
For convenience `Table.toString()` automatically registers the table under a unique name in its `TableEnvironment` and returns the name. Hence, `Table` objects can be directly inlined into SQL queries (by string concatenation) as shown in the examples below.
@@ -113,8 +113,12 @@ Flink parses SQL using [Apache Calcite](https://calcite.apache.org/docs/referenc
The following BNF-grammar describes the superset of supported SQL features in batch and streaming queries. The [Operations](#operations) section shows examples for the supported features and indicates which features are only supported for batch or streaming queries.
-```
+{% highlight sql %}
+insert:
+ INSERT INTO tableReference
+ query
+
query:
values
| {
@@ -139,7 +143,8 @@ select:
[ WHERE booleanExpression ]
[ GROUP BY { groupItem [, groupItem ]* } ]
[ HAVING booleanExpression ]
-
+ [ WINDOW windowName AS windowSpec [, windowName AS windowSpec ]* ]
+
selectWithoutFrom:
SELECT [ ALL | DISTINCT ]
{ * | projectItem [, projectItem ]* }
@@ -176,11 +181,22 @@ groupItem:
| ROLLUP '(' expression [, expression ]* ')'
| GROUPING SETS '(' groupItem [, groupItem ]* ')'
-insert:
- INSERT INTO tableReference
- query
+windowRef:
+ windowName
+ | windowSpec
+
+windowSpec:
+ [ windowName ]
+ '('
+ [ ORDER BY orderItem [, orderItem ]* ]
+ [ PARTITION BY expression [, expression ]* ]
+ [
+ RANGE numericOrIntervalExpression {PRECEDING}
+ | ROWS numericExpression {PRECEDING}
+ ]
+ ')'
-```
+{% endhighlight %}
Flink SQL uses a lexical policy for identifier (table, attribute, function names) similar to Java:
@@ -302,6 +318,13 @@ SELECT COUNT(amount) OVER (
ORDER BY proctime
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
FROM Orders
+
+SELECT COUNT(amount) OVER w, SUM(amount) OVER w
+FROM Orders
+WINDOW w AS (
+ PARTITION BY user
+ ORDER BY proctime
+ ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
{% endhighlight %}
</td>
</tr>
@@ -388,12 +411,14 @@ GROUP BY users
SELECT *
FROM Orders INNER JOIN Product ON Orders.productId = Product.id
{% endhighlight %}
-<p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
+ <p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
<td><strong>Outer Equi-join</strong><br>
<span class="label label-primary">Batch</span>
+ <span class="label label-primary">Streaming</span>
+ <span class="label label-info">Result Updating</span>
</td>
<td>
<p>Currently, only equi-joins are supported, i.e., joins that have at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.</p>
@@ -408,6 +433,7 @@ FROM Orders RIGHT JOIN Product ON Orders.productId = Product.id
SELECT *
FROM Orders FULL OUTER JOIN Product ON Orders.productId = Product.id
{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
@@ -456,19 +482,19 @@ FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
<span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
</td>
<td>
- <p>UDTFs must be registered in the TableEnvironment. See the <a href="udfs.html">UDF documentation</a> for details on how to specify and register UDTFs. </p>
- <p>Inner Join</p>
+ <p>UDTFs must be registered in the TableEnvironment. See the <a href="udfs.html">UDF documentation</a> for details on how to specify and register UDTFs. </p>
+ <p>Inner Join</p>
{% highlight sql %}
SELECT users, tag
FROM Orders, LATERAL TABLE(unnest_udtf(tags)) t AS tag
{% endhighlight %}
- <p>Left Outer Join</p>
+ <p>Left Outer Join</p>
{% highlight sql %}
SELECT users, tag
FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) t AS tag ON TRUE
{% endhighlight %}
-<p><b>Note:</b> Currently, only literal <code>TRUE</code> is supported as predicate for a left outer join against a lateral table.</p>
+ <p><b>Note:</b> Currently, only literal <code>TRUE</code> is supported as predicate for a left outer join against a lateral table.</p>
</td>
</tr>
</tbody>
@@ -549,10 +575,10 @@ FROM (
<tr>
<td>
<strong>In</strong><br>
- <span class="label label-primary">Batch</span>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
</td>
<td>
- Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.
+ <p>Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.</p>
{% highlight sql %}
SELECT user, amount
FROM Orders
@@ -560,6 +586,25 @@ WHERE product IN (
SELECT product FROM NewProducts
)
{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <strong>Exists</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Returns true if the sub-query returns at least one row. Only supported if the operation can be rewritten in a join and group operation.</p>
+{% highlight sql %}
+SELECT user, amount
+FROM Orders
+WHERE product EXISTS (
+ SELECT product FROM NewProducts
+)
+{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
</tbody>
@@ -1081,6 +1126,7 @@ EXISTS (sub-query)
</td>
<td>
<p>Returns TRUE if <i>sub-query</i> returns at least one row. Only supported if the operation can be rewritten in a join and group operation.</p>
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
@@ -1091,7 +1137,8 @@ value IN (sub-query)
{% endhighlight %}
</td>
<td>
- <p>Returns TRUE if <i>value</i> is equal to a row returned by sub-query. This operation is not supported in a streaming environment yet.</p>
+ <p>Returns TRUE if <i>value</i> is equal to a row returned by sub-query.</p>
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
@@ -1102,7 +1149,8 @@ value NOT IN (sub-query)
{% endhighlight %}
</td>
<td>
- <p>Returns TRUE if <i>value</i> is not equal to every row returned by sub-query. This operation is not supported in a streaming environment yet.</p>
+ <p>Returns TRUE if <i>value</i> is not equal to every row returned by sub-query.</p>
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
@@ -1989,6 +2037,17 @@ EXTRACT(timeintervalunit FROM temporal)
</td>
</tr>
+ <tr>
+ <td>
+ {% highlight text %}
+YEAR(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the year from a SQL date. Equivalent to <code>EXTRACT(YEAR FROM date)</code>. E.g. <code>YEAR(DATE '1994-09-27')</code> leads to 1994.</p>
+ </td>
+ </tr>
+
<tr>
<td>
{% highlight text %}
@@ -2018,7 +2077,95 @@ QUARTER(date)
{% endhighlight %}
</td>
<td>
- <p>Returns the quarter of a year from a SQL date. E.g. <code>QUARTER(DATE '1994-09-27')</code> leads to 3.</p>
+ <p>Returns the quarter of a year from a SQL date (an integer between 1 and 4). Equivalent to <code>EXTRACT(QUARTER FROM date)</code>. E.g. <code>QUARTER(DATE '1994-09-27')</code> leads to 3. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+MONTH(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the month of a year from a SQL date (an integer between 1 and 12). Equivalent to <code>EXTRACT(MONTH FROM date)</code>. E.g. <code>MONTH(DATE '1994-09-27')</code> leads to 9. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+WEEK(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the week of a year from a SQL date (an integer between 1 and 53). Equivalent to <code>EXTRACT(WEEK FROM date)</code>. E.g. <code>WEEK(DATE '1994-09-27')</code> leads to 39. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+DAYOFYEAR(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the day of a year from a SQL date (an integer between 1 and 366). Equivalent to <code>EXTRACT(DOY FROM date)</code>. E.g. <code>DAYOFYEAR(DATE '1994-09-27')</code> leads to 270. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+DAYOFMONTH(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the day of a month from a SQL date (an integer between 1 and 31). Equivalent to <code>EXTRACT(DAY FROM date)</code>. E.g. <code>DAYOFMONTH(DATE '1994-09-27')</code> leads to 27. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+DAYOFWEEK(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the day of a week from a SQL date (an integer between 1 and 7; Sunday = 1). Equivalent to <code>EXTRACT(DOW FROM date)</code>. E.g. <code>DAYOFWEEK(DATE '1994-09-27')</code> leads to 3. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+HOUR(timestamp)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the hour of a day from a SQL timestamp (an integer between 0 and 23). Equivalent to <code>EXTRACT(HOUR FROM timestamp)</code>. E.g. <code>HOUR(TIMESTAMP '1994-09-27 13:14:15')</code> leads to 13. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+MINUTE(timestamp)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the minute of an hour from a SQL timestamp (an integer between 0 and 59). Equivalent to <code>EXTRACT(MINUTE FROM timestamp)</code>. E.g. <code>MINUTE(TIMESTAMP '1994-09-27 13:14:15')</code> leads to 14. </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SECOND(timestamp)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the second of a minute from a SQL timestamp (an integer between 0 and 59). Equivalent to <code>EXTRACT(SECOND FROM timestamp)</code>. E.g. <code>SECOND(TIMESTAMP '1994-09-27 13:14:15')</code> leads to 15. </p>
</td>
</tr>
@@ -2075,7 +2222,7 @@ COUNT(value [, value]* )
{% endhighlight %}
</td>
<td>
- <p>Returns the number of input rows for which <i>value</i> is not null.</p>
+ <p>Returns the number of input rows for which <i>value</i> is not null. Use <code>COUNT(DISTINCT value)</code> for the number of unique values in the column or expression.</p>
</td>
</tr>
@@ -2108,7 +2255,7 @@ SUM(numeric)
{% endhighlight %}
</td>
<td>
- <p>Returns the sum of <i>numeric</i> across all input values.</p>
+ <p>Returns the sum of <i>numeric</i> across all input values. Use <code>SUM(DISTINCT value)</code> for the sum of unique values in the column or expression.</p>
</td>
</tr>
@@ -2180,8 +2327,8 @@ VAR_SAMP(value)
<tr>
<td>
{% highlight text %}
- COLLECT(value)
- {% endhighlight %}
+COLLECT(value)
+{% endhighlight %}
</td>
<td>
<p>Returns a multiset of the <i>value</i>s. null input <i>value</i> will be ignored. Return an empty multiset if only null values are added. </p>
@@ -2419,7 +2566,7 @@ MD5(string)
{% endhighlight %}
</td>
<td>
- <p>Returns the MD5 hash of the string argument as a string of 32 hexadecimal digits; null if <i>string</i> is null.</p>
+ <p>Returns the MD5 hash of the <i>string</i> argument as a string of 32 hexadecimal digits; null if <i>string</i> is null.</p>
</td>
</tr>
@@ -2430,10 +2577,21 @@ SHA1(string)
{% endhighlight %}
</td>
<td>
- <p>Returns the SHA-1 hash of the string argument as a string of 40 hexadecimal digits; null if <i>string</i> is null.</p>
+ <p>Returns the SHA-1 hash of the <i>string</i> argument as a string of 40 hexadecimal digits; null if <i>string</i> is null.</p>
</td>
</tr>
+ <tr>
+ <td>
+ {% highlight text %}
+SHA224(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-224 hash of the <i>string</i> argument as a string of 56 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
<tr>
<td>
{% highlight text %}
@@ -2441,7 +2599,41 @@ SHA256(string)
{% endhighlight %}
</td>
<td>
- <p>Returns the SHA-256 hash of the string argument as a string of 64 hexadecimal digits; null if <i>string</i> is null.</p>
+ <p>Returns the SHA-256 hash of the <i>string</i> argument as a string of 64 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SHA384(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-384 hash of the <i>string</i> argument as a string of 96 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SHA512(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-512 hash of the <i>string</i> argument as a string of 128 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SHA2(string, hashLength)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the hash using the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, or SHA-512). The first argument <i>string</i> is the string to be hashed. <i>hashLength</i> is the bit length of the result (either 224, 256, 384, or 512). Returns <i>null</i> if <i>string</i> or <i>hashLength</i> is <i>null</i>.
+ </p>
</td>
</tr>
</tbody>
@@ -2453,7 +2645,6 @@ The following functions are not supported yet:
- Binary string operators and functions
- System functions
-- Distinct aggregate functions like COUNT DISTINCT
{% top %}
@@ -2464,7 +2655,7 @@ Although not every SQL feature is implemented yet, some string combinations are
{% highlight sql %}
-A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, ALWAYS, AND, ANY, ARE, ARRAY, AS, ASC, ASENSITIVE, ASSERTION, ASSIGNMENT, ASYMMETRIC, AT, ATOMIC, ATTRIBUTE, ATTRIBUTES, AUTHORIZATION, AVG, BEFORE, BEGIN, BERNOULLI, BETWEEN, BIGINT, BINARY, BIT, BLOB, BOOLEAN, BOTH, BREADTH, BY, C, CALL, CALLED, CARDINALITY, CASCADE, CASCADED, CASE, CAST, CATALOG, CATALOG_NAME, CEIL, CEILING, CENTURY, CHAIN, CHAR, CHARACTER, CHARACTERISTICTS, CHARACTERS, CHARACTER_LENGTH, CHARACTER_SET_CATALOG, CHARACTER_SET_NAME, CHARACTER_SET_SCHEMA, CHAR_LENGTH, CHECK, CLASS_ORIGIN, CLOB, CLOSE, COALESCE, COBOL, COLLATE, COLLATION, COLLATION_CATALOG, COLLATION_NAME, COLLATION_SCHEMA, COLLECT, COLUMN, COLUMN_NAME, COMMAND_FUNCTION, COMMAND_FUNCTION_CODE, COMMIT, COMMITTED, CONDITION, CONDITION_NUMBER, CONNECT, CONNECTION, CONNECTION_NAME, CONSTRAINT, CONSTRAINTS, CONSTRAINT_CATALOG, CONSTRAINT_NAME, CONSTRAINT_SCHEMA, CONSTRUCTOR, CONTAINS, CONTINUE, CONVERT, CORR, CORRESPONDING, COUNT, COVAR_POP, COVAR_SAMP, CREATE, CROSS, CUBE, CUME_DIST, CURRENT, CURRENT_CATALOG, CURRENT_DATE, CURRENT_DEFAULT_TRANSFORM_GROUP, CURRENT_PATH, CURRENT_ROLE, CURRENT_SCHEMA, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_TRANSFORM_GROUP_FOR_TYPE, CURRENT_USER, CURSOR, CURSOR_NAME, CYCLE, DATA, DATABASE, DATE, DATETIME_INTERVAL_CODE, DATETIME_INTERVAL_PRECISION, DAY, DEALLOCATE, DEC, DECADE, DECIMAL, DECLARE, DEFAULT, DEFAULTS, DEFERRABLE, DEFERRED, DEFINED, DEFINER, DEGREE, DELETE, DENSE_RANK, DEPTH, DEREF, DERIVED, DESC, DESCRIBE, DESCRIPTION, DESCRIPTOR, DETERMINISTIC, DIAGNOSTICS, DISALLOW, DISCONNECT, DISPATCH, DISTINCT, DOMAIN, DOUBLE, DOW, DOY, DROP, DYNAMIC, DYNAMIC_FUNCTION, DYNAMIC_FUNCTION_CODE, EACH, ELEMENT, ELSE, END, END-EXEC, EPOCH, EQUALS, ESCAPE, EVERY, EXCEPT, EXCEPTION, EXCLUDE, EXCLUDING, EXEC, EXECUTE, EXISTS, EXP, EXPLAIN, EXTEND, EXTERNAL, EXTRACT, FALSE, FETCH, FILTER, FINAL, FIRST, FIRST_VALUE, FLOAT, FLOOR, FOLLOWING, FOR, FOREIGN, FORTRAN, FOUND, FRAC_SECOND, FREE, FROM, FULL, FUNCTION, FUSION, G, GENERAL, GENERATED, GET, GLOBAL, GO, GOTO, GRANT, GRANTED, GROUP, GROUPING, HAVING, HIERARCHY, HOLD, HOUR, IDENTITY, IMMEDIATE, IMPLEMENTATION, IMPORT, IN, INCLUDING, INCREMENT, INDICATOR, INITIALLY, INNER, INOUT, INPUT, INSENSITIVE, INSERT, INSTANCE, INSTANTIABLE, INT, INTEGER, INTERSECT, INTERSECTION, INTERVAL, INTO, INVOKER, IS, ISOLATION, JAVA, JOIN, K, KEY, KEY_MEMBER, KEY_TYPE, LABEL, LANGUAGE, LARGE, LAST, LAST_VALUE, LATERAL, LEADING, LEFT, LENGTH, LEVEL, LIBRARY, LIKE, LIMIT, LN, LOCAL, LOCALTIME, LOCALTIMESTAMP, LOCATOR, LOWER, M, MAP, MATCH, MATCHED, MAX, MAXVALUE, MEMBER, MERGE, MESSAGE_LENGTH, MESSAGE_OCTET_LENGTH, MESSAGE_TEXT, METHOD, MICROSECOND, MILLENNIUM, MIN, MINUTE, MINVALUE, MOD, MODIFIES, MODULE, MONTH, MORE, MULTISET, MUMPS, NAME, NAMES, NATIONAL, NATURAL, NCHAR, NCLOB, NESTING, NEW, NEXT, NO, NONE, NORMALIZE, NORMALIZED, NOT, NULL, NULLABLE, NULLIF, NULLS, NUMBER, NUMERIC, OBJECT, OCTETS, OCTET_LENGTH, OF, OFFSET, OLD, ON, ONLY, OPEN, OPTION, OPTIONS, OR, ORDER, ORDERING, ORDINALITY, OTHERS, OUT, OUTER, OUTPUT, OVER, OVERLAPS, OVERLAY, OVERRIDING, PAD, PARAMETER, PARAMETER_MODE, PARAMETER_NAME, PARAMETER_ORDINAL_POSITION, PARAMETER_SPECIFIC_CATALOG, PARAMETER_SPECIFIC_NAME, PARAMETER_SPECIFIC_SCHEMA, PARTIAL, PARTITION, PASCAL, PASSTHROUGH, PATH, PERCENTILE_CONT, PERCENTILE_DISC, PERCENT_RANK, PLACING, PLAN, PLI, POSITION, POWER, PRECEDING, PRECISION, PREPARE, PRESERVE, PRIMARY, PRIOR, PRIVILEGES, PROCEDURE, PUBLIC, QUARTER, RANGE, RANK, READ, READS, REAL, RECURSIVE, REF, REFERENCES, REFERENCING, REGR_AVGX, REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE, REGR_SXX, REGR_SXY, REGR_SYY, RELATIVE, RELEASE, REPEATABLE, RESET, RESTART, RESTRICT, RESULT, RETURN, RETURNED_CARDINALITY, RETURNED_LENGTH, RETURNED_OCTET_LENGTH, RETURNED_SQLSTATE, RETURNS, REVOKE, RIGHT, ROLE, ROLLBACK, ROLLUP, ROUTINE, ROUTINE_CATALOG, ROUTINE_NAME, ROUTINE_SCHEMA, ROW, ROWS, ROW_COUNT, ROW_NUMBER, SAVEPOINT, SCALE, SCHEMA, SCHEMA_NAME, SCOPE, SCOPE_CATALOGS, SCOPE_NAME, SCOPE_SCHEMA, SCROLL, SEARCH, SECOND, SECTION, SECURITY, SELECT, SELF, SENSITIVE, SEQUENCE, SERIALIZABLE, SERVER, SERVER_NAME, SESSION, SESSION_USER, SET, SETS, SIMILAR, SIMPLE, SIZE, SMALLINT, SOME, SOURCE, SPACE, SPECIFIC, SPECIFICTYPE, SPECIFIC_NAME, SQL, SQLEXCEPTION, SQLSTATE, SQLWARNING, SQL_TSI_DAY, SQL_TSI_FRAC_SECOND, SQL_TSI_HOUR, SQL_TSI_MICROSECOND, SQL_TSI_MINUTE, SQL_TSI_MONTH, SQL_TSI_QUARTER, SQL_TSI_SECOND, SQL_TSI_WEEK, SQL_TSI_YEAR, SQRT, START, STATE, STATEMENT, STATIC, STDDEV_POP, STDDEV_SAMP, STREAM, STRUCTURE, STYLE, SUBCLASS_ORIGIN, SUBMULTISET, SUBSTITUTE, SUBSTRING, SUM, SYMMETRIC, SYSTEM, SYSTEM_USER, TABLE, TABLESAMPLE, TABLE_NAME, TEMPORARY, THEN, TIES, TIME, TIMESTAMP, TIMESTAMPADD, TIMESTAMPDIFF, TIMEZONE_HOUR, TIMEZONE_MINUTE, TINYINT, TO, TOP_LEVEL_COUNT, TRAILING, TRANSACTION, TRANSACTIONS_ACTIVE, TRANSACTIONS_COMMITTED, TRANSACTIONS_ROLLED_BACK, TRANSFORM, TRANSFORMS, TRANSLATE, TRANSLATION, TREAT, TRIGGER, TRIGGER_CATALOG, TRIGGER_NAME, TRIGGER_SCHEMA, TRIM, TRUE, TYPE, UESCAPE, UNBOUNDED, UNCOMMITTED, UNDER, UNION, UNIQUE, UNKNOWN, UNNAMED, UNNEST, UPDATE, UPPER, UPSERT, USAGE, USER, USER_DEFINED_TYPE_CATALOG, USER_DEFINED_TYPE_CODE, USER_DEFINED_TYPE_NAME, USER_DEFINED_TYPE_SCHEMA, USING, VALUE, VALUES, VARBINARY, VARCHAR, VARYING, VAR_POP, VAR_SAMP, VERSION, VIEW, WEEK, WHEN, WHENEVER, WHERE, WIDTH_BUCKET, WINDOW, WITH, WITHIN, WITHOUT, WORK, WRAPPER, WRITE, XML, YEAR, ZONE
+A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, ALWAYS, AND, ANY, ARE, ARRAY, AS, ASC, ASENSITIVE, ASSERTION, ASSIGNMENT, ASYMMETRIC, AT, ATOMIC, ATTRIBUTE, ATTRIBUTES, AUTHORIZATION, AVG, BEFORE, BEGIN, BERNOULLI, BETWEEN, BIGINT, BINARY, BIT, BLOB, BOOLEAN, BOTH, BREADTH, BY, C, CALL, CALLED, CARDINALITY, CASCADE, CASCADED, CASE, CAST, CATALOG, CATALOG_NAME, CEIL, CEILING, CENTURY, CHAIN, CHAR, CHARACTER, CHARACTERISTICS, CHARACTERS, CHARACTER_LENGTH, CHARACTER_SET_CATALOG, CHARACTER_SET_NAME, CHARACTER_SET_SCHEMA, CHAR_LENGTH, CHECK, CLASS_ORIGIN, CLOB, CLOSE, COALESCE, COBOL, COLLATE, COLLATION, COLLATION_CATALOG, COLLATION_NAME, COLLATION_SCHEMA, COLLECT, COLUMN, COLUMN_NAME, COMMAND_FUNCTION, COMMAND_FUNCTION_CODE, COMMIT, COMMITTED, CONDITION, CONDITION_NUMBER, CONNECT, CONNECTION, CONNECTION_NAME, CONSTRAINT, CONSTRAINTS, CONSTRAINT_CATALOG, CONSTRAINT_NAME, CONSTRAINT_SCHEMA, CONSTRUCTOR, CONTAINS, CONTINUE, CONVERT, CORR, CORRESPONDING, COUNT, COVAR_POP, COVAR_SAMP, CREATE, CROSS, CUBE, CUME_DIST, CURRENT, CURRENT_CATALOG, CURRENT_DATE, CURRENT_DEFAULT_TRANSFORM_GROUP, CURRENT_PATH, CURRENT_ROLE, CURRENT_SCHEMA, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_TRANSFORM_GROUP_FOR_TYPE, CURRENT_USER, CURSOR, CURSOR_NAME, CYCLE, DATA, DATABASE, DATE, DATETIME_INTERVAL_CODE, DATETIME_INTERVAL_PRECISION, DAY, DEALLOCATE, DEC, DECADE, DECIMAL, DECLARE, DEFAULT, DEFAULTS, DEFERRABLE, DEFERRED, DEFINED, DEFINER, DEGREE, DELETE, DENSE_RANK, DEPTH, DEREF, DERIVED, DESC, DESCRIBE, DESCRIPTION, DESCRIPTOR, DETERMINISTIC, DIAGNOSTICS, DISALLOW, DISCONNECT, DISPATCH, DISTINCT, DOMAIN, DOUBLE, DOW, DOY, DROP, DYNAMIC, DYNAMIC_FUNCTION, DYNAMIC_FUNCTION_CODE, EACH, ELEMENT, ELSE, END, END-EXEC, EPOCH, EQUALS, ESCAPE, EVERY, EXCEPT, EXCEPTION, EXCLUDE, EXCLUDING, EXEC, EXECUTE, EXISTS, EXP, EXPLAIN, EXTEND, EXTERNAL, EXTRACT, FALSE, FETCH, FILTER, FINAL, FIRST, FIRST_VALUE, FLOAT, FLOOR, FOLLOWING, FOR, FOREIGN, FORTRAN, FOUND, FRAC_SECOND, FREE, FROM, FULL, FUNCTION, FUSION, G, GENERAL, GENERATED, GET, GLOBAL, GO, GOTO, GRANT, GRANTED, GROUP, GROUPING, HAVING, HIERARCHY, HOLD, HOUR, IDENTITY, IMMEDIATE, IMPLEMENTATION, IMPORT, IN, INCLUDING, INCREMENT, INDICATOR, INITIALLY, INNER, INOUT, INPUT, INSENSITIVE, INSERT, INSTANCE, INSTANTIABLE, INT, INTEGER, INTERSECT, INTERSECTION, INTERVAL, INTO, INVOKER, IS, ISOLATION, JAVA, JOIN, K, KEY, KEY_MEMBER, KEY_TYPE, LABEL, LANGUAGE, LARGE, LAST, LAST_VALUE, LATERAL, LEADING, LEFT, LENGTH, LEVEL, LIBRARY, LIKE, LIMIT, LN, LOCAL, LOCALTIME, LOCALTIMESTAMP, LOCATOR, LOWER, M, MAP, MATCH, MATCHED, MAX, MAXVALUE, MEMBER, MERGE, MESSAGE_LENGTH, MESSAGE_OCTET_LENGTH, MESSAGE_TEXT, METHOD, MICROSECOND, MILLENNIUM, MIN, MINUTE, MINVALUE, MOD, MODIFIES, MODULE, MONTH, MORE, MULTISET, MUMPS, NAME, NAMES, NATIONAL, NATURAL, NCHAR, NCLOB, NESTING, NEW, NEXT, NO, NONE, NORMALIZE, NORMALIZED, NOT, NULL, NULLABLE, NULLIF, NULLS, NUMBER, NUMERIC, OBJECT, OCTETS, OCTET_LENGTH, OF, OFFSET, OLD, ON, ONLY, OPEN, OPTION, OPTIONS, OR, ORDER, ORDERING, ORDINALITY, OTHERS, OUT, OUTER, OUTPUT, OVER, OVERLAPS, OVERLAY, OVERRIDING, PAD, PARAMETER, PARAMETER_MODE, PARAMETER_NAME, PARAMETER_ORDINAL_POSITION, PARAMETER_SPECIFIC_CATALOG, PARAMETER_SPECIFIC_NAME, PARAMETER_SPECIFIC_SCHEMA, PARTIAL, PARTITION, PASCAL, PASSTHROUGH, PATH, PERCENTILE_CONT, PERCENTILE_DISC, PERCENT_RANK, PLACING, PLAN, PLI, POSITION, POWER, PRECEDING, PRECISION, PREPARE, PRESERVE, PRIMARY, PRIOR, PRIVILEGES, PROCEDURE, PUBLIC, QUARTER, RANGE, RANK, READ, READS, REAL, RECURSIVE, REF, REFERENCES, REFERENCING, REGR_AVGX, REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE, REGR_SXX, REGR_SXY, REGR_SYY, RELATIVE, RELEASE, REPEATABLE, RESET, RESTART, RESTRICT, RESULT, RETURN, RETURNED_CARDINALITY, RETURNED_LENGTH, RETURNED_OCTET_LENGTH, RETURNED_SQLSTATE, RETURNS, REVOKE, RIGHT, ROLE, ROLLBACK, ROLLUP, ROUTINE, ROUTINE_CATALOG, ROUTINE_NAME, ROUTINE_SCHEMA, ROW, ROWS, ROW_COUNT, ROW_NUMBER, SAVEPOINT, SCALE, SCHEMA, SCHEMA_NAME, SCOPE, SCOPE_CATALOGS, SCOPE_NAME, SCOPE_SCHEMA, SCROLL, SEARCH, SECOND, SECTION, SECURITY, SELECT, SELF, SENSITIVE, SEQUENCE, SERIALIZABLE, SERVER, SERVER_NAME, SESSION, SESSION_USER, SET, SETS, SIMILAR, SIMPLE, SIZE, SMALLINT, SOME, SOURCE, SPACE, SPECIFIC, SPECIFICTYPE, SPECIFIC_NAME, SQL, SQLEXCEPTION, SQLSTATE, SQLWARNING, SQL_TSI_DAY, SQL_TSI_FRAC_SECOND, SQL_TSI_HOUR, SQL_TSI_MICROSECOND, SQL_TSI_MINUTE, SQL_TSI_MONTH, SQL_TSI_QUARTER, SQL_TSI_SECOND, SQL_TSI_WEEK, SQL_TSI_YEAR, SQRT, START, STATE, STATEMENT, STATIC, STDDEV_POP, STDDEV_SAMP, STREAM, STRUCTURE, STYLE, SUBCLASS_ORIGIN, SUBMULTISET, SUBSTITUTE, SUBSTRING, SUM, SYMMETRIC, SYSTEM, SYSTEM_USER, TABLE, TABLESAMPLE, TABLE_NAME, TEMPORARY, THEN, TIES, TIME, TIMESTAMP, TIMESTAMPADD, TIMESTAMPDIFF, TIMEZONE_HOUR, TIMEZONE_MINUTE, TINYINT, TO, TOP_LEVEL_COUNT, TRAILING, TRANSACTION, TRANSACTIONS_ACTIVE, TRANSACTIONS_COMMITTED, TRANSACTIONS_ROLLED_BACK, TRANSFORM, TRANSFORMS, TRANSLATE, TRANSLATION, TREAT, TRIGGER, TRIGGER_CATALOG, TRIGGER_NAME, TRIGGER_SCHEMA, TRIM, TRUE, TYPE, UESCAPE, UNBOUNDED, UNCOMMITTED, UNDER, UNION, UNIQUE, UNKNOWN, UNNAMED, UNNEST, UPDATE, UPPER, UPSERT, USAGE, USER, USER_DEFINED_TYPE_CATALOG, USER_DEFINED_TYPE_CODE, USER_DEFINED_TYPE_NAME, USER_DEFINED_TYPE_SCHEMA, USING, VALUE, VALUES, VARBINARY, VARCHAR, VARYING, VAR_POP, VAR_SAMP, VERSION, VIEW, WEEK, WHEN, WHENEVER, WHERE, WIDTH_BUCKET, WINDOW, WITH, WITHIN, WITHOUT, WORK, WRAPPER, WRITE, XML, YEAR, ZONE
{% endhighlight %}
@@ -2556,9 +2747,6 @@ A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, A
<tr><td>{% highlight text %}%W{% endhighlight %}</td>
<td>Weekday name (<code>Sunday</code> .. <code>Saturday</code>)</td>
</tr>
- <tr><td>{% highlight text %}%W{% endhighlight %}</td>
- <td>Weekday name (<code>Sunday</code> .. <code>Saturday</code>)</td>
- </tr>
<tr><td>{% highlight text %}%w{% endhighlight %}</td>
<td>Day of the week (<code>0</code> .. <code>6</code>), where Sunday is the first day of the week</td>
</tr>
diff --git a/docs/dev/table/sqlClient.md b/docs/dev/table/sqlClient.md
new file mode 100644
index 00000000000..24af6557dc6
--- /dev/null
+++ b/docs/dev/table/sqlClient.md
@@ -0,0 +1,641 @@
+---
+title: "SQL Client"
+nav-parent_id: tableapi
+nav-pos: 100
+is_beta: true
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+Flink’s Table & SQL API makes it possible to work with queries written in the SQL language, but these queries need to be embedded within a table program that is written in either Java or Scala. Moreover, these programs need to be packaged with a build tool before being submitted to a cluster. This more or less limits the usage of Flink to Java/Scala programmers.
+
+The *SQL Client* aims to provide an easy way of writing, debugging, and submitting table programs to a Flink cluster without a single line of Java or Scala code. The *SQL Client CLI* allows for retrieving and visualizing real-time results from the running distributed application on the command line.
+
+<a href="{{ site.baseurl }}/fig/sql_client_demo.gif"><img class="offset" src="{{ site.baseurl }}/fig/sql_client_demo.gif" alt="Animated demo of the Flink SQL Client CLI running table programs on a cluster" width="80%" /></a>
+
+<span class="label label-danger">Attention</span> The SQL Client is in an early development phase. Even though the application is not production-ready yet, it can be a quite useful tool for prototyping and playing around with Flink SQL. In the future, the community plans to extend its functionality by providing a REST-based [SQL Client Gateway](sqlClient.html#limitations--future).
+
+* This will be replaced by the TOC
+{:toc}
+
+Getting Started
+---------------
+
+This section describes how to setup and run your first Flink SQL program from the command-line.
+
+The SQL Client is bundled in the regular Flink distribution and thus runnable out-of-the-box. It requires only a running Flink cluster where table programs can be executed. For more information about setting up a Flink cluster see the [Cluster & Deployment]({{ site.baseurl }}/ops/deployment/cluster_setup.html) part. If you simply want to try out the SQL Client, you can also start a local cluster with one worker using the following command:
+
+{% highlight bash %}
+./bin/start-cluster.sh
+{% endhighlight %}
+
+### Starting the SQL Client CLI
+
+The SQL Client scripts are also located in the binary directory of Flink. [In the future](sqlClient.html#limitations--future), a user will have two possibilities of starting the SQL Client CLI either by starting an embedded standalone process or by connecting to a remote SQL Client Gateway. At the moment only the `embedded` mode is supported. You can start the CLI by calling:
+
+{% highlight bash %}
+./bin/sql-client.sh embedded
+{% endhighlight %}
+
+By default, the SQL Client will read its configuration from the environment file located in `./conf/sql-client-defaults.yaml`. See the [configuration part](sqlClient.html#environment-files) for more information about the structure of environment files.
+
+### Running SQL Queries
+
+Once the CLI has been started, you can use the `HELP` command to list all available SQL statements. For validating your setup and cluster connection, you can enter your first SQL query and press the `Enter` key to execute it:
+
+{% highlight sql %}
+SELECT 'Hello World'
+{% endhighlight %}
+
+This query requires no table source and produces a single row result. The CLI will retrieve results from the cluster and visualize them. You can close the result view by pressing the `Q` key.
+
+The CLI supports **two modes** for maintaining and visualizing results.
+
+The **table mode** materializes results in memory and visualizes them in a regular, paginated table representation. It can be enabled by executing the following command in the CLI:
+
+{% highlight text %}
+SET execution.result-mode=table
+{% endhighlight %}
+
+The **changelog mode** does not materialize results and visualizes the result stream that is produced by a [continuous query](streaming.html#dynamic-tables--continuous-queries) consisting of insertions (`+`) and retractions (`-`).
+
+{% highlight text %}
+SET execution.result-mode=changelog
+{% endhighlight %}
+
+You can use the following query to see both result modes in action:
+
+{% highlight sql %}
+SELECT name, COUNT(*) AS cnt FROM (VALUES ('Bob'), ('Alice'), ('Greg'), ('Bob')) AS NameTable(name) GROUP BY name
+{% endhighlight %}
+
+This query performs a bounded word count example.
+
+In *changelog mode*, the visualized changelog should be similar to:
+
+{% highlight text %}
++ Bob, 1
++ Alice, 1
++ Greg, 1
+- Bob, 1
++ Bob, 2
+{% endhighlight %}
+
+In *table mode*, the visualized result table is continuously updated until the table program ends with:
+
+{% highlight text %}
+Bob, 2
+Alice, 1
+Greg, 1
+{% endhighlight %}
+
+The [configuration section](sqlClient.html#configuration) explains how to read from table sources and configure other table program properties.
+
+{% top %}
+
+Configuration
+-------------
+
+The SQL Client can be started with the following optional CLI commands. They are discussed in detail in the subsequent paragraphs.
+
+{% highlight text %}
+./bin/sql-client.sh embedded --help
+
+Mode "embedded" submits Flink jobs from the local machine.
+
+ Syntax: embedded [OPTIONS]
+ "embedded" mode options:
+ -d,--defaults <environment file> The environment properties with which
+ every new session is initialized.
+ Properties might be overwritten by
+ session properties.
+ -e,--environment <environment file> The environment properties to be
+ imported into the session. It might
+ overwrite default environment
+ properties.
+ -h,--help Show the help message with
+ descriptions of all options.
+ -j,--jar <JAR file> A JAR file to be imported into the
+ session. The file might contain
+ user-defined classes needed for the
+ execution of statements such as
+ functions, table sources, or sinks.
+ Can be used multiple times.
+ -l,--library <JAR directory> A JAR file directory with which every
+ new session is initialized. The files
+ might contain user-defined classes
+ needed for the execution of
+ statements such as functions, table
+ sources, or sinks. Can be used
+ multiple times.
+ -s,--session <session identifier> The identifier for a session.
+ 'default' is the default identifier.
+{% endhighlight %}
+
+{% top %}
+
+### Environment Files
+
+A SQL query needs a configuration environment in which it is executed. The so-called *environment files* define available table sources and sinks, external catalogs, user-defined functions, and other properties required for execution and deployment.
+
+Every environment file is a regular [YAML file](http://yaml.org/). An example of such a file is presented below.
+
+{% highlight yaml %}
+# Define table sources here.
+
+tables:
+ - name: MyTableName
+ type: source
+ schema:
+ - name: MyField1
+ type: INT
+ - name: MyField2
+ type: VARCHAR
+ connector:
+ type: filesystem
+ path: "/path/to/something.csv"
+ format:
+ type: csv
+ fields:
+ - name: MyField1
+ type: INT
+ - name: MyField2
+ type: VARCHAR
+ line-delimiter: "\n"
+ comment-prefix: "#"
+
+# Execution properties allow for changing the behavior of a table program.
+
+execution:
+ type: streaming # required: execution mode either 'batch' or 'streaming'
+ result-mode: table # required: either 'table' or 'changelog'
+ time-characteristic: event-time # optional: 'processing-time' or 'event-time' (default)
+ parallelism: 1 # optional: Flink's parallelism (1 by default)
+ periodic-watermarks-interval: 200 # optional: interval for periodic watermarks (200 ms by default)
+ max-parallelism: 16 # optional: Flink's maximum parallelism (128 by default)
+ min-idle-state-retention: 0 # optional: table program's minimum idle state time
+ max-idle-state-retention: 0 # optional: table program's maximum idle state time
+
+# Deployment properties allow for describing the cluster to which table programs are submitted to.
+
+deployment:
+ response-timeout: 5000
+{% endhighlight %}
+
+This configuration:
+
+- defines an environment with a table source `MyTableName` that reads from a CSV file,
+- specifies a parallelism of 1 for queries executed in this streaming environment,
+- specifies an event-time characteristic, and
+- runs queries in the `table` result mode.
+
+Depending on the use case, a configuration can be split into multiple files. Therefore, environment files can be created for general purposes (*defaults environment file* using `--defaults`) as well as on a per-session basis (*session environment file* using `--environment`). Every CLI session is initialized with the default properties followed by the session properties. For example, the defaults environment file could specify all table sources that should be available for querying in every session whereas the session environment file only declares a specific state retention time and parallelism. Both default and session environment files can be passed when starting the CLI application. If no default environment file has been specified, the SQL Client searches for `./conf/sql-client-defaults.yaml` in Flink's configuration directory.
+
+<span class="label label-danger">Attention</span> Properties that have been set within a CLI session (e.g. using the `SET` command) have highest precedence:
+
+{% highlight text %}
+CLI commands > session environment file > defaults environment file
+{% endhighlight %}
+
+Queries that are executed in a batch environment, can only be retrieved using the `table` result mode.
+
+{% top %}
+
+### Dependencies
+
+The SQL Client does not require to setup a Java project using Maven or SBT. Instead, you can pass the dependencies as regular JAR files that get submitted to the cluster. You can either specify each JAR file separately (using `--jar`) or define entire library directories (using `--library`). For connectors to external systems (such as Apache Kafka) and corresponding data formats (such as JSON), Flink provides **ready-to-use JAR bundles**. These JAR files are suffixed with `sql-jar` and can be downloaded for each release from the Maven central repository.
+
+{% if site.is_stable %}
+
+#### Connectors
+
+| Name | Version | Download |
+| :---------------- | :------------ | :--------------------- |
+| Filesystem | | Built-in |
+| Apache Kafka | 0.9 | [Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.9{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.9{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar) |
+| Apache Kafka | 0.10 | [Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.10{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.10{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar) |
+| Apache Kafka | 0.11 | [Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.11{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.11{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar) |
+
+#### Formats
+
+| Name | Download |
+| :---------------- | :--------------------- |
+| CSV | Built-in |
+| JSON | [Download](http://central.maven.org/maven2/org/apache/flink/flink-json/{{site.version}}/flink-json-{{site.version}}-sql-jar.jar) |
+| Apache Avro | [Download](http://central.maven.org/maven2/org/apache/flink/flink-avro/{{site.version}}/flink-avro-{{site.version}}-sql-jar.jar) |
+
+{% endif %}
+
+{% top %}
+
+Table Sources
+-------------
+
+Sources are defined using a set of [YAML properties](http://yaml.org/). Similar to a SQL `CREATE TABLE` statement you define the name of the table, the final schema of the table, connector, and a data format if necessary. Additionally, you have to specify its type (source, sink, or both).
+
+{% highlight yaml %}
+name: MyTable # required: string representing the table name
+type: source # required: currently only 'source' is supported
+schema: ... # required: final table schema
+connector: ... # required: connector configuration
+format: ... # optional: format that depends on the connector
+{% endhighlight %}
+
+<span class="label label-danger">Attention</span> Not every combination of connector and format is supported. Internally, your YAML file is translated into a set of string-based properties by which the SQL Client tries to resolve a matching table source. If a table source can be resolved also depends on the JAR files available in the classpath.
+
+The following example shows an environment file that defines a table source reading JSON data from Apache Kafka. All properties are explained in the following subsections.
+
+{% highlight yaml %}
+tables:
+ - name: TaxiRides
+ type: source
+ schema:
+ - name: rideId
+ type: LONG
+ - name: lon
+ type: FLOAT
+ - name: lat
+ type: FLOAT
+ - name: rowTime
+ type: TIMESTAMP
+ rowtime:
+ timestamps:
+ type: "from-field"
+ from: "rideTime"
+ watermarks:
+ type: "periodic-bounded"
+ delay: "60000"
+ - name: procTime
+ type: TIMESTAMP
+ proctime: true
+ connector:
+ property-version: 1
+ type: kafka
+ version: 0.11
+ topic: TaxiRides
+ startup-mode: earliest-offset
+ properties:
+ - key: zookeeper.connect
+ value: localhost:2181
+ - key: bootstrap.servers
+ value: localhost:9092
+ - key: group.id
+ value: testGroup
+ format:
+ property-version: 1
+ type: json
+ schema: "ROW(rideId LONG, lon FLOAT, lat FLOAT, rideTime TIMESTAMP)"
+{% endhighlight %}
+
+The resulting schema of the `TaxiRide` table contains most of the fields of the JSON schema. Furthermore, it adds a rowtime attribute `rowTime` and processing-time attribute `procTime`. Both `connector` and `format` allow to define a property version (which is currently version `1`) for future backwards compatibility.
+
+{% top %}
+
+### Schema Properties
+
+The schema allows for describing the final appearance of a table. It specifies the final name, final type, and the origin of a field. The origin of a field might be important if the name of the field should differ from the input format. For instance, a field `name&field` should reference `nameField` from an Avro format.
+
+{% highlight yaml %}
+schema:
+ - name: MyField1
+ type: ...
+ - name: MyField2
+ type: ...
+ - name: MyField3
+ type: ...
+{% endhighlight %}
+
+For *each field*, the following properties can be used:
+
+{% highlight yaml %}
+name: ... # required: final name of the field
+type: ... # required: final type of the field represented as a type string
+proctime: ... # optional: boolean flag whether this field should be a processing-time attribute
+rowtime: ... # optional: wether this field should be a event-time attribute
+from: ... # optional: original field in the input that is referenced/aliased by this field
+{% endhighlight %}
+
+#### Type Strings
+
+The following type strings are supported for being defined in an environment file:
+
+{% highlight yaml %}
+VARCHAR
+BOOLEAN
+TINYINT
+SMALLINT
+INT
+BIGINT
+FLOAT
+DOUBLE
+DECIMAL
+DATE
+TIME
+TIMESTAMP
+ROW(fieldtype, ...) # unnamed row; e.g. ROW(VARCHAR, INT) that is mapped to Flink's RowTypeInfo
+ # with indexed fields names f0, f1, ...
+ROW(fieldname fieldtype, ...) # named row; e.g., ROW(myField VARCHAR, myOtherField INT) that
+ # is mapped to Flink's RowTypeInfo
+POJO(class) # e.g., POJO(org.mycompany.MyPojoClass) that is mapped to Flink's PojoTypeInfo
+ANY(class) # e.g., ANY(org.mycompany.MyClass) that is mapped to Flink's GenericTypeInfo
+ANY(class, serialized) # used for type information that is not supported by Flink's Table & SQL API
+{% endhighlight %}
+
+#### Rowtime Properties
+
+In order to control the event-time behavior for table sources, the SQL Client provides predefined timestamp extractors and watermark strategies. For more information about time handling in Flink and especially event-time, we recommend the general [event-time section](streaming.html#time-attributes).
+
+The following timestamp extractors are supported:
+
+{% highlight yaml %}
+# Converts an existing BIGINT or TIMESTAMP field in the input into the rowtime attribute.
+rowtime:
+ timestamps:
+ type: from-field
+ from: ... # required: original field name in the input
+
+# Converts the assigned timestamps from a DataStream API record into the rowtime attribute
+# and thus preserves the assigned timestamps from the source.
+rowtime:
+ timestamps:
+ type: from-source
+{% endhighlight %}
+
+The following watermark strategies are supported:
+
+{% highlight yaml %}
+# Sets a watermark strategy for ascending rowtime attributes. Emits a watermark of the maximum
+# observed timestamp so far minus 1. Rows that have a timestamp equal to the max timestamp
+# are not late.
+rowtime:
+ watermarks:
+ type: periodic-ascending
+
+# Sets a built-in watermark strategy for rowtime attributes which are out-of-order by a bounded time interval.
+# Emits watermarks which are the maximum observed timestamp minus the specified delay.
+rowtime:
+ watermarks:
+ type: periodic-bounded
+ delay: ... # required: delay in milliseconds
+
+# Sets a built-in watermark strategy which indicates the watermarks should be preserved from the
+# underlying DataStream API and thus preserves the assigned watermarks from the source.
+rowtime:
+ watermarks:
+ type: from-source
+{% endhighlight %}
+
+{% top %}
+
+### Connector Properties
+
+Flink provides a set of connectors that can be defined in the environment file.
+
+<span class="label label-danger">Attention</span> Currently, connectors can only be used as table sources not sinks.
+
+#### Filesystem Connector
+
+The filesystem connector allows for reading from a local or distributed filesystem. A filesystem can be defined as:
+
+{% highlight yaml %}
+connector:
+ type: filesystem
+ path: "file:///path/to/whatever" # required
+{% endhighlight %}
+
+Currently, only files with CSV format can be read from a filesystem. The filesystem connector is included in Flink and does not require an additional JAR file.
+
+#### Kafka Connector
+
+The Kafka connector allows for reading from a Apache Kafka topic. It can be defined as follows:
+
+{% highlight yaml %}
+connector:
+ type: kafka
+ version: 0.11 # required: valid connector versions are "0.8", "0.9", "0.10", and "0.11"
+ topic: ... # required: topic name from which the table is read
+ startup-mode: ... # optional: valid modes are "earliest-offset", "latest-offset",
+ # "group-offsets", or "specific-offsets"
+ specific-offsets: # optional: used in case of startup mode with specific offsets
+ - partition: 0
+ offset: 42
+ - partition: 1
+ offset: 300
+ properties: # optional: connector specific properties
+ - key: zookeeper.connect
+ value: localhost:2181
+ - key: bootstrap.servers
+ value: localhost:9092
+ - key: group.id
+ value: testGroup
+{% endhighlight %}
+
+Make sure to download the [Kafka SQL JAR](sqlClient.html#dependencies) file and pass it to the SQL Client.
+
+{% top %}
+
+### Format Properties
+
+Flink provides a set of formats that can be defined in the environment file.
+
+#### CSV Format
+
+The CSV format allows to read comma-separated rows. Currently, this is only supported for the filesystem connector.
+
+{% highlight yaml %}
+format:
+ type: csv
+ fields: # required: format fields
+ - name: field1
+ type: VARCHAR
+ - name: field2
+ type: TIMESTAMP
+ field-delimiter: "," # optional: string delimiter "," by default
+ line-delimiter: "\n" # optional: string delimiter "\n" by default
+ quote-character: '"' # optional: single character for string values, empty by default
+ comment-prefix: '#' # optional: string to indicate comments, empty by default
+ ignore-first-line: false # optional: boolean flag to ignore the first line, by default it is not skipped
+ ignore-parse-errors: true # optional: skip records with parse error instead to fail by default
+{% endhighlight %}
+
+The CSV format is included in Flink and does not require an additional JAR file.
+
+#### JSON Format
+
+The JSON format allows to read and write JSON data that corresponds to a given format schema. The format schema can be defined either as a Flink [type string](sqlClient.html#type-strings), as a JSON schema, or derived from the desired table schema. A type string enables a more SQL-like definition and mapping to the corresponding SQL data types. The JSON schema allows for more complex and nested structures.
+
+If the format schema is equal to the table schema, the schema can also be automatically derived. This allows for defining schema information only once. The names, types, and field order of the format are determined by the table's schema. Time attributes are ignored. A `from` definition in the table schema is interpreted as a field renaming in the format.
+
+{% highlight yaml %}
+format:
+ type: json
+ fail-on-missing-field: true # optional: flag whether to fail if a field is missing or not
+
+ # required: define the schema either by using a type string which parses numbers to corresponding types
+ schema: "ROW(lon FLOAT, rideTime TIMESTAMP)"
+
+ # or by using a JSON schema which parses to DECIMAL and TIMESTAMP
+ json-schema: >
+ {
+ type: 'object',
+ properties: {
+ lon: {
+ type: 'number'
+ },
+ rideTime: {
+ type: 'string',
+ format: 'date-time'
+ }
+ }
+ }
+
+ # or use the tables schema
+ derive-schema: true
+{% endhighlight %}
+
+The following table shows the mapping of JSON schema types to Flink SQL types:
+
+| JSON schema | Flink SQL |
+| :-------------------------------- | :---------------------- |
+| `object` | `ROW` |
+| `boolean` | `BOOLEAN` |
+| `array` | `ARRAY[_]` |
+| `number` | `DECIMAL` |
+| `integer` | `DECIMAL` |
+| `string` | `VARCHAR` |
+| `string` with `format: date-time` | `TIMESTAMP` |
+| `string` with `format: date` | `DATE` |
+| `string` with `format: time` | `TIME` |
+| `string` with `encoding: base64` | `ARRAY[TINYINT]` |
+| `null` | `NULL` (unsupported yet)|
+
+
+Currently, Flink supports only a subset of the [JSON schema specification](http://json-schema.org/) `draft-07`. Union types (as well as `allOf`, `anyOf`, `not`) are not supported yet. `oneOf` and arrays of types are only supported for specifying nullability.
+
+Simple references that link to a common definition in the document are supported as shown in the more complex example below:
+
+{% highlight json %}
+{
+ "definitions": {
+ "address": {
+ "type": "object",
+ "properties": {
+ "street_address": {
+ "type": "string"
+ },
+ "city": {
+ "type": "string"
+ },
+ "state": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "street_address",
+ "city",
+ "state"
+ ]
+ }
+ },
+ "type": "object",
+ "properties": {
+ "billing_address": {
+ "$ref": "#/definitions/address"
+ },
+ "shipping_address": {
+ "$ref": "#/definitions/address"
+ },
+ "optional_address": {
+ "oneOf": [
+ {
+ "type": "null"
+ },
+ {
+ "$ref": "#/definitions/address"
+ }
+ ]
+ }
+ }
+}
+{% endhighlight %}
+
+Make sure to download the [JSON SQL JAR](sqlClient.html#dependencies) file and pass it to the SQL Client.
+
+#### Apache Avro Format
+
+The [Apache Avro](https://avro.apache.org/) format allows to read and write Avro data that corresponds to a given format schema. The format schema can be defined either as a fully qualified class name of an Avro specific record or as an Avro schema string. If a class name is used, the class must be available in the classpath during runtime.
+
+{% highlight yaml %}
+format:
+ type: avro
+
+ # required: define the schema either by using an Avro specific record class
+ record-class: "org.organization.types.User"
+
+ # or by using an Avro schema
+ avro-schema: >
+ {
+ "type": "record",
+ "name": "test",
+ "fields" : [
+ {"name": "a", "type": "long"},
+ {"name": "b", "type": "string"}
+ ]
+ }
+{% endhighlight %}
+
+Avro types are mapped to the corresponding SQL data types. Union types are only supported for specifying nullability otherwise they are converted to an `ANY` type. The following table shows the mapping:
+
+| Avro schema | Flink SQL |
+| :------------------------------------------ | :---------------------- |
+| `record` | `ROW` |
+| `enum` | `VARCHAR` |
+| `array` | `ARRAY[_]` |
+| `map` | `MAP[VARCHAR, _]` |
+| `union` | non-null type or `ANY` |
+| `fixed` | `ARRAY[TINYINT]` |
+| `string` | `VARCHAR` |
+| `bytes` | `ARRAY[TINYINT]` |
+| `int` | `INT` |
+| `long` | `BIGINT` |
+| `float` | `FLOAT` |
+| `double` | `DOUBLE` |
+| `boolean` | `BOOLEAN` |
+| `int` with `logicalType: date` | `DATE` |
+| `int` with `logicalType: time-millis` | `TIME` |
+| `int` with `logicalType: time-micros` | `INT` |
+| `long` with `logicalType: timestamp-millis` | `TIMESTAMP` |
+| `long` with `logicalType: timestamp-micros` | `BIGINT` |
+| `bytes` with `logicalType: decimal` | `DECIMAL` |
+| `fixed` with `logicalType: decimal` | `DECIMAL` |
+| `null` | `NULL` (unsupported yet)|
+
+Avro uses [Joda-Time](http://www.joda.org/joda-time/) for representing logical date and time types in specific record classes. The Joda-Time dependency is not part of Flink's SQL JAR distribution. Therefore, make sure that Joda-Time is in your classpath together with your specific record class during runtime. Avro formats specified via a schema string do not require Joda-Time to be present.
+
+Make sure to download the [Apache Avro SQL JAR](sqlClient.html#dependencies) file and pass it to the SQL Client.
+
+{% top %}
+
+Limitations & Future
+--------------------
+
+The current SQL Client implementation is in a very early development stage and might change in the future as part of the bigger Flink Improvement Proposal 24 ([FLIP-24](https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client)). Feel free to join the discussion and open issue about bugs and features that you find useful.
+
+{% top %}
diff --git a/docs/dev/table/streaming.md b/docs/dev/table/streaming.md
index dc0fdf86d7d..d0fea47cbfd 100644
--- a/docs/dev/table/streaming.md
+++ b/docs/dev/table/streaming.md
@@ -84,13 +84,13 @@ The following figure visualizes the relationship of streams, dynamic tables, and
In the following, we will explain the concepts of dynamic tables and continuous queries with a stream of click events that have the following schema:
-```
+{% highlight plain %}
[
user: VARCHAR, // the name of the user
cTime: TIMESTAMP, // the time when the URL was accessed
url: VARCHAR // the URL that was accessed by the user
]
-```
+{% endhighlight %}
### Defining a Table on a Stream
@@ -507,7 +507,7 @@ StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// obtain query configuration from TableEnvironment
StreamQueryConfig qConfig = tableEnv.queryConfig();
// set query parameters
-qConfig.withIdleStateRetentionTime(Time.hours(12));
+qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(24));
// define query
Table result = ...
@@ -531,7 +531,7 @@ val tableEnv = TableEnvironment.getTableEnvironment(env)
// obtain query configuration from TableEnvironment
val qConfig: StreamQueryConfig = tableEnv.queryConfig
// set query parameters
-qConfig.withIdleStateRetentionTime(Time.hours(12))
+qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(24))
// define query
val result: Table = ???
@@ -557,9 +557,9 @@ Many queries aggregate or join records on one or more key attributes. When such
For example the following query computes the number of clicks per session.
-```
+{% highlight sql %}
SELECT sessionId, COUNT(*) FROM clicks GROUP BY sessionId;
-```
+{% endhighlight %}
The `sessionId` attribute is used as a grouping key and the continuous query maintains a count for each `sessionId` it observes. The `sessionId` attribute is evolving over time and `sessionId` values are only active until the session ends, i.e., for a limited period of time. However, the continuous query cannot know about this property of `sessionId` and expects that every `sessionId` value can occur at any point of time. It maintains a count for each observed `sessionId` value. Consequently, the total state size of the query is continuously growing as more and more `sessionId` values are observed.
@@ -579,10 +579,8 @@ The parameters are specified as follows:
StreamQueryConfig qConfig = ...
-// set idle state retention time: min = 12 hour, max = 16 hours
-qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(16));
-// set idle state retention time. min = max = 12 hours
-qConfig.withIdleStateRetentionTime(Time.hours(12);
+// set idle state retention time: min = 12 hours, max = 24 hours
+qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(24));
{% endhighlight %}
</div>
@@ -591,16 +589,14 @@ qConfig.withIdleStateRetentionTime(Time.hours(12);
val qConfig: StreamQueryConfig = ???
-// set idle state retention time: min = 12 hour, max = 16 hours
-qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(16))
-// set idle state retention time. min = max = 12 hours
-qConfig.withIdleStateRetentionTime(Time.hours(12)
+// set idle state retention time: min = 12 hours, max = 24 hours
+qConfig.withIdleStateRetentionTime(Time.hours(12), Time.hours(24))
{% endhighlight %}
</div>
</div>
-Configuring different minimum and maximum idle state retention times is more efficient because it reduces the internal book-keeping of a query for when to remove state.
+Cleaning up state requires additional bookkeeping which becomes less expensive for larger differences of `minTime` and `maxTime`. The difference between `minTime` and `maxTime` must be at least 5 minutes.
{% top %}
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
index 69aaeea756e..2f651454903 100644
--- a/docs/dev/table/tableApi.md
+++ b/docs/dev/table/tableApi.md
@@ -506,8 +506,10 @@ Table result = left.join(right).where("a = d").select("a, b, e");
<tr>
<td>
- <strong>Outer Joins</strong><br>
+ <strong>Outer Join</strong><br>
<span class="label label-primary">Batch</span>
+ <span class="label label-primary">Streaming</span>
+ <span class="label label-info">Result Updating</span>
</td>
<td>
<p>Similar to SQL LEFT/RIGHT/FULL OUTER JOIN clauses. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
@@ -519,6 +521,7 @@ Table leftOuterResult = left.leftOuterJoin(right, "a = d").select("a, b, e");
Table rightOuterResult = left.rightOuterJoin(right, "a = d").select("a, b, e");
Table fullOuterResult = left.fullOuterJoin(right, "a = d").select("a, b, e");
{% endhighlight %}
+<p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
@@ -610,6 +613,7 @@ Table result = orders
<td>
<strong>Inner Join</strong><br>
<span class="label label-primary">Batch</span>
+ <span class="label label-primary">Streaming</span>
</td>
<td>
<p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined through join operator or using a where or filter operator.</p>
@@ -618,12 +622,15 @@ val left = ds1.toTable(tableEnv, 'a, 'b, 'c)
val right = ds2.toTable(tableEnv, 'd, 'e, 'f)
val result = left.join(right).where('a === 'd).select('a, 'b, 'e)
{% endhighlight %}
+<p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
<td>
- <strong>Outer Joins</strong><br>
+ <strong>Outer Join</strong><br>
<span class="label label-primary">Batch</span>
+ <span class="label label-primary">Streaming</span>
+ <span class="label label-info">Result Updating</span>
</td>
<td>
<p>Similar to SQL LEFT/RIGHT/FULL OUTER JOIN clauses. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
@@ -635,6 +642,7 @@ val leftOuterResult = left.leftOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
val rightOuterResult = left.rightOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
val fullOuterResult = left.fullOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
{% endhighlight %}
+<p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
@@ -815,7 +823,7 @@ Table result = left.minusAll(right);
<tr>
<td>
<strong>In</strong><br>
- <span class="label label-primary">Batch</span>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
</td>
<td>
<p>Similar to a SQL IN clause. In returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.</p>
@@ -830,6 +838,8 @@ Table result = left.select("a, b, c").where("a.in(" + right + ")");
tableEnv.registerTable("RightTable", right);
Table result = left.select("a, b, c").where("a.in(RightTable)");
{% endhighlight %}
+
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
</tbody>
@@ -940,7 +950,7 @@ val result = left.minusAll(right)
<tr>
<td>
<strong>In</strong><br>
- <span class="label label-primary">Batch</span>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
</td>
<td>
<p>Similar to a SQL IN clause. In returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.</p>
@@ -949,6 +959,7 @@ val left = ds1.toTable(tableEnv, 'a, 'b, 'c)
val right = ds2.toTable(tableEnv, 'a)
val result = left.select('a, 'b, 'c).where('a.in(right))
{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
@@ -1791,7 +1802,32 @@ ANY.in(TABLE)
{% endhighlight %}
</td>
<td>
- <p>Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression. Note: This operation is not supported in a streaming environment yet.</p>
+ <p>Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.</p>
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.between(lowerBound, upperBound)
+ {% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is between <i>lowerBound</i> and <i>upperBound</i> (both inclusive). False otherwise. The parameters must be numeric types or identical comparable types.
+ </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.notBetween(lowerBound, upperBound)
+ {% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is not between <i>lowerBound</i> and <i>upperBound</i> (both inclusive). False otherwise. The parameters must be numeric types or identical comparable types.
+ </p>
</td>
</tr>
@@ -3083,7 +3119,18 @@ STRING.sha1()
</td>
</tr>
- <tr>
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.sha224()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-224 hash of the string argument as a string of 56 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
<td>
{% highlight java %}
STRING.sha256()
@@ -3094,6 +3141,40 @@ STRING.sha256()
</td>
</tr>
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.sha384()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-384 hash of the string argument as a string of 96 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.sha512()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-512 hash of the string argument as a string of 128 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.sha2(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the hash using the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, or SHA-512). The first argument <i>string</i> is the string to be hashed. <i>hashLength</i> is the bit length of the result (either 224, 256, 384, or 512). Returns <i>null</i> if <i>string</i> or <i>hashLength</i> is <i>null</i>.
+ </p>
+ </td>
+ </tr>
+
</tbody>
</table>
@@ -3287,6 +3368,31 @@ ANY.in(TABLE)
</td>
<td>
<p>Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression. Note: This operation is not supported in a streaming environment yet.</p>
+ <p><b>Note:</b> For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY.between(lowerBound, upperBound)
+ {% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is between <i>lowerBound</i> and <i>upperBound</i> (both inclusive). False otherwise. The parameters must be numeric types or identical comparable types.
+ </p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY.notBetween(lowerBound, upperBound)
+ {% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is not between <i>lowerBound</i> and <i>upperBound</i> (both inclusive). False otherwise. The parameters must be numeric types or identical comparable types.
+ </p>
</td>
</tr>
@@ -4529,7 +4635,18 @@ STRING.sha1()
</td>
</tr>
- <tr>
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.sha224()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-224 hash of the string argument as a string of 56 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
<td>
{% highlight scala %}
STRING.sha256()
@@ -4539,7 +4656,40 @@ STRING.sha256()
<p>Returns the SHA-256 hash of the string argument as a string of 64 hexadecimal digits; null if <i>string</i> is null.</p>
</td>
</tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.sha384()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-384 hash of the string argument as a string of 96 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.sha512()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the SHA-512 hash of the string argument as a string of 128 hexadecimal digits; null if <i>string</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.sha2(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the hash using the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, or SHA-512). The first argument <i>string</i> is the string to be hashed. <i>hashLength</i> is the bit length of the result (either 224, 256, 384, or 512). Returns <i>null</i> if <i>string</i> or <i>hashLength</i> is <i>null</i>.
+ </p>
+ </td>
+ </tr>
</tbody>
</table>
diff --git a/docs/dev/table/udfs.md b/docs/dev/table/udfs.md
index 02c047e67db..7dce8009777 100644
--- a/docs/dev/table/udfs.md
+++ b/docs/dev/table/udfs.md
@@ -292,7 +292,7 @@ optionally implemented. While some of these methods allow the system more effici
**The following methods of `AggregateFunction` are required depending on the use case:**
- `retract()` is required for aggregations on bounded `OVER` windows.
-- `merge()` is required for many batch aggreagtions and session window aggregations.
+- `merge()` is required for many batch aggregations and session window aggregations.
- `resetAccumulator()` is required for many batch aggregations.
All methods of `AggregateFunction` must be declared as `public`, not `static` and named exactly as the names mentioned above. The methods `createAccumulator`, `getValue`, `getResultType`, and `getAccumulatorType` are defined in the `AggregateFunction` abstract class, while others are contracted methods. In order to define a aggregate function, one has to extend the base class `org.apache.flink.table.functions.AggregateFunction` and implement one (or more) `accumulate` methods. The method `accumulate` can be overloaded with different parameter types and supports variable arguments.
diff --git a/docs/fig/sql_client_demo.gif b/docs/fig/sql_client_demo.gif
new file mode 100644
index 00000000000..bb0554e4d0e
Binary files /dev/null and b/docs/fig/sql_client_demo.gif differ
diff --git a/docs/index.md b/docs/index.md
index 1ba8f07c100..419bb397d8a 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -31,7 +31,7 @@ Apache Flink is an open source platform for distributed stream and batch data pr
## First Steps
-- **Concepts**: Start with the basic concepts of Flink's [Dataflow Programming Model](concepts/programming-model.html) and [Distributed Runtime Environment](concepts/runtime.html). This will help you understand other parts of the documentation, including the setup and programming guides. We recommended you read these sections first.
+- **Concepts**: Start with the basic concepts of Flink's [Dataflow Programming Model](concepts/programming-model.html) and [Distributed Runtime Environment](concepts/runtime.html). This will help you understand other parts of the documentation, including the setup and programming guides. We recommend you read these sections first.
- **Quickstarts**: [Run an example program](quickstart/setup_quickstart.html) on your local machine or [study some examples](examples/index.html).
@@ -41,12 +41,12 @@ Apache Flink is an open source platform for distributed stream and batch data pr
Before putting your Flink job into production, read the [Production Readiness Checklist](ops/production_ready.html).
-## Migration Guide
+## Release Notes
-For users of earlier versions of Apache Flink, we recommend the [API migration guide](dev/migration.html).
-While all parts of the API that were marked as public and stable are still supported (the public API is backwards compatible), we suggest migrating applications to the newer interfaces where applicable.
+Release notes cover important changes between Flink versions. Please carefully read these notes if you plan to upgrade your Flink setup to a later version.
-For users that plan to upgrade a Flink system in production, we recommend reading the guide on [upgrading Apache Flink](ops/upgrading.html).
+* [Release notes for Flink 1.6](release-notes/flink-1.6.html).
+* [Release notes for Flink 1.5](release-notes/flink-1.5.html).
## External Resources
diff --git a/docs/internals/ide_setup.md b/docs/internals/ide_setup.md
index 340544d7346..c59ea8d39f1 100644
--- a/docs/internals/ide_setup.md
+++ b/docs/internals/ide_setup.md
@@ -81,7 +81,7 @@ to enable support for Scala projects and files:
files for the IDE to work with but without installing libraries.
8. Build the Project (Build -> Make Project)
-### Checkstyle
+### Checkstyle For Java
IntelliJ supports checkstyle within the IDE using the Checkstyle-IDEA plugin.
1. Install the "Checkstyle-IDEA" plugin from the IntelliJ plugin repository.
@@ -112,6 +112,10 @@ clicking the "Check Module" button. The scan should report no errors.
which include `flink-core`, `flink-optimizer`, and `flink-runtime`.
Nevertheless please make sure that code you add/modify in these modules still conforms to the checkstyle rules.
+### Checkstyle For Scala
+
+Enable scalastyle in Intellij by selecting Settings -> Editor -> Inspections, then searching for "Scala style inspections". Also Place `"tools/maven/scalastyle_config.xml"` in the `"<root>/.idea"` or `"<root>/project"` directory.
+
## Eclipse
**NOTE:** From our experience, this setup does not work with Flink
diff --git a/docs/monitoring/application_profiling.md b/docs/monitoring/application_profiling.md
index 721bc31a2d2..b9cec5b69b0 100644
--- a/docs/monitoring/application_profiling.md
+++ b/docs/monitoring/application_profiling.md
@@ -40,17 +40,17 @@ Java Flight Recorder is a profiling and event collection framework built into th
is an advanced set of tools that enables efficient and detailed analysis of the extensive of data collected by Java
Flight Recorder. Example configuration:
-~~~
+{% highlight yaml %}
env.java.opts: "-XX:+UnlockCommercialFeatures -XX:+UnlockDiagnosticVMOptions -XX:+FlightRecorder -XX:+DebugNonSafepoints -XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=${FLINK_LOG_PREFIX}.jfr"
-~~~
+{% endhighlight %}
# Profiling with JITWatch
[JITWatch](https://github.com/AdoptOpenJDK/jitwatch/wiki) is a log analyser and visualizer for the Java HotSpot JIT
compiler used to inspect inlining decisions, hot methods, bytecode, and assembly. Example configuration:
-~~~
+{% highlight yaml %}
env.java.opts: "-XX:+UnlockDiagnosticVMOptions -XX:+TraceClassLoading -XX:+LogCompilation -XX:LogFile=${FLINK_LOG_PREFIX}.jit -XX:+PrintAssembly"
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/monitoring/checkpoint_monitoring.md b/docs/monitoring/checkpoint_monitoring.md
index 25ed3dde74d..0a66ca5f162 100644
--- a/docs/monitoring/checkpoint_monitoring.md
+++ b/docs/monitoring/checkpoint_monitoring.md
@@ -68,10 +68,10 @@ The checkpoint history keeps statistics about recently triggered checkpoints, in
You can configure the number of recent checkpoints that are remembered for the history via the following configuration key. The default is `10`.
-```sh
+{% highlight yaml %}
# Number of recent checkpoints that are remembered
jobmanager.web.checkpoints.history: 15
-```
+{% endhighlight %}
### Summary Tab
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
index 187ffafcfdf..fe2c36d1f14 100644
--- a/docs/monitoring/debugging_classloading.md
+++ b/docs/monitoring/debugging_classloading.md
@@ -30,26 +30,44 @@ under the License.
When running Flink applications, the JVM will load various classes over time.
These classes can be divided into two domains:
- - The **Flink Framework** domain: This includes all code in the `/lib` directory in the Flink directory.
- By default these are the classes of Apache Flink and its core dependencies.
+ - The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
+ in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
- - The **User Code** domain: These are all classes that are included in the JAR file submitted via the CLI or web interface.
- That includes the job's classes, and all libraries and connectors that are put into the uber JAR.
+ - The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
+ (via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
+What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
+processes first, and the submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
+or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
-The class loading behaves slightly different for various Flink setups:
+In the following are some more details about the different deployment modes:
-**Standalone**
+**Standalone Session**
+
+When starting a Flink cluster as a standalone session, the JobManagers and TaskManagers are started with the Flink framework classes in the
+Java classpath. The classes from all jobs/applications that are submitted against the session (via REST / CLI) are loaded *dynamically*.
+
+<!--
+**Docker Containers with Flink-as-a-Library**
+
+If you package a Flink job/application such that your application treats Flink like a library (Flink JobManager/TaskManager daemons as spawned as needed),
+then typically all classes are in the *application classpath*. This is the recommended way for container-based setups where the container is specifically
+created for an job/application and will contain the job/application's jar files.
+
+-->
+
+**Docker / Kubernetes Sessions**
+
+Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
+behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
-When starting a Flink cluster, the JobManagers and TaskManagers are started with the Flink framework classes in the
-classpath. The classes from all jobs that are submitted against the cluster are loaded *dynamically*.
**YARN**
YARN classloading differs between single job deployments and sessions:
- - When submitting a Flink job directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in their classpath.
+ - When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
+ JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
That means that there is *no dynamic classloading* involved in that case.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
@@ -58,20 +76,36 @@ YARN classloading differs between single job deployments and sessions:
**Mesos**
Mesos setups following [this documentation](../ops/deployment/mesos.html) currently behave very much like the a
-YARN session: The TaskManager and JobManager processes are started with the Flink framework classes in classpath, job
+YARN session: The TaskManager and JobManager processes are started with the Flink framework classes in the Java classpath, job
classes are loaded dynamically when the jobs are submitted.
-## Configuring ClassLoader Resolution Order
-Flink uses a hierarchy of ClassLoaders for loading classes from the user-code jar(s). The user-code
-ClassLoader has a reference to the parent ClassLoader, which is the default Java ClassLoader in most
-cases. By default, Java ClassLoaders will first look for classes in the parent ClassLoader and then in
-the child ClassLoader for cases where we have a hierarchy of ClassLoaders. This is problematic if you
-have in your user jar a version of a library that conflicts with a version that comes with Flink. You can
-change this behaviour by configuring the ClassLoader resolution order via
-`classloader.resolve-order: child-first` in the Flink config. However, Flink classes will still
-be resolved through the parent ClassLoader first, although you can also configure this via
-`classloader.parent-first-patterns` (see [config](../ops/config.html))
+## Inverted Class Loading and ClassLoader Resolution Order
+
+In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
+for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+cases.
+
+By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded user code.
+
+The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
+errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
+classes (Flink's core or one of its dependencies can use a different copy than the user code).
+In most cases, this work well and no additional configuration from the user is needed.
+
+However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
+You can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+(from Flink's default `child-first`).
+
+Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
+are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
+[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
+[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
## Avoiding Dynamic Classloading
@@ -103,26 +137,22 @@ and access the user code class loader via `getRuntimeContext().getUserCodeClassL
## X cannot be cast to X exceptions
-When you see an exception in the style `com.foo.X cannot be cast to com.foo.X`, it means that multiple versions of the class
-`com.foo.X` have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.
-
-The reason is in most cases that an object of the `com.foo.X` class loaded from a previous execution attempt is still cached somewhere,
-and picked up by a restarted task/operator that reloaded the code. Note that this is again only possible in deployments that use
-dynamic class loading.
+In setups with dynamic classloading, you may see an exception in the style `com.foo.X cannot be cast to com.foo.X`.
+This means that multiple versions of the class `com.foo.X` have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.
-Common causes of cached object instances:
+One common reason is that a library is not compatible with Flink's *inverted classloading* approach. You can turn off inverted classloading
+to verify this (set [`classloader.resolve-order: parent-first`](../ops/config.html#classloader-resolve-order) in the Flink config) or exclude
+the library from inverted classloading (set [`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional)
+in the Flink config).
- - When using *Apache Avro*: The *SpecificDatumReader* caches instances of records. Avoid using `SpecificData.INSTANCE`. See also
- [this discussion](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-get-help-on-ClassCastException-when-re-submitting-a-job-tp10972p11133.html)
-
- - Using certain serialization frameworks for cloning objects (such as *Apache Avro*)
-
- - Interning objects (for example via Guava's Interners)
+Another cause can be cached object instances, as produced by some libraries like *Apache Avro*, or by interning objects (for example via Guava's Interners).
+The solution here is to either have a setup without any dynamic classloading, or to make sure that the respective library is fully part of the dynamically loaded code.
+The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
## Unloading of Dynamically Loaded Classes
-All scenarios that involve dynamic class loading (i.e., standalone, sessions, mesos, ...) rely on classes being *unloaded* again.
+All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).
@@ -141,15 +171,14 @@ Common causes for class leaks and suggested fixes:
## Resolving Dependency Conflicts with Flink using the maven-shade-plugin.
-Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often `IllegalAccessExceptions` or `NoSuchMethodError` are the result.
+A way to address dependency conflicts from the application developer's side is to avoid exposing dependencies by *shading them away*.
-Through Hadoop, Flink for example depends on the `aws-sdk` library or on `protobuf-java`. If your user code is using these libraries and you run into issues we recommend relocating the dependency in your user code jar.
-
-Apache Maven offers the [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-plugin/), which allows one to change the package of a class *after* compiling it (so the code you are writing is not affected by the shading). For example if you have the `com.amazonaws` packages from the aws sdk in your user code jar, the shade plugin would relocate them into the `org.myorg.shaded.com.amazonaws` package, so that your code is calling your aws sdk version.
+Apache Maven offers the [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-plugin/), which allows one to change the package of a
+class *after* compiling it (so the code you are writing is not affected by the shading). For example if you have the `com.amazonaws` packages from
+the aws sdk in your user code jar, the shade plugin would relocate them into the `org.myorg.shaded.com.amazonaws` package, so that your code is calling your aws sdk version.
This documentation page explains [relocating classes using the shade plugin](https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html).
-
-Note that some of Flink's dependencies, such as `guava` are shaded away by the maintainers of Flink, so users usually don't have to worry about it.
+Note that most of Flink's dependencies, such as `guava`, `netty`, `jackson`, etc. are shaded away by the maintainers of Flink, so users usually don't have to worry about it.
{% top %}
diff --git a/docs/monitoring/historyserver.md b/docs/monitoring/historyserver.md
index 9c68e652508..7f8f737e220 100644
--- a/docs/monitoring/historyserver.md
+++ b/docs/monitoring/historyserver.md
@@ -35,10 +35,10 @@ The HistoryServer allows you to query the status and statistics of completed job
After you have configured the HistoryServer *and* JobManager, you start and stop the HistoryServer via its corresponding startup script:
-```sh
+{% highlight shell %}
# Start or stop the HistoryServer
bin/historyserver.sh (start|start-foreground|stop)
-```
+{% endhighlight %}
By default, this server binds to `localhost` and listens at port `8082`.
@@ -52,22 +52,22 @@ The configuration keys `jobmanager.archive.fs.dir` and `historyserver.archive.fs
The archiving of completed jobs happens on the JobManager, which uploads the archived job information to a file system directory. You can configure the directory to archive completed jobs in `flink-conf.yaml` by setting a directory via `jobmanager.archive.fs.dir`.
-```sh
+{% highlight yaml %}
# Directory to upload completed job information
jobmanager.archive.fs.dir: hdfs:///completed-jobs
-```
+{% endhighlight %}
**HistoryServer**
The HistoryServer can be configured to monitor a comma-separated list of directories in via `historyserver.archive.fs.dir`. The configured directories are regularly polled for new archives; the polling interval can be configured via `historyserver.archive.fs.refresh-interval`.
-```sh
+{% highlight yaml %}
# Monitor the following directories for completed jobs
historyserver.archive.fs.dir: hdfs:///completed-jobs
# Refresh every 10 seconds
historyserver.archive.fs.refresh-interval: 10000
-```
+{% endhighlight %}
The contained archives are downloaded and cached in the local filesystem. The local directory for this is configured via `historyserver.web.tmpdir`.
diff --git a/docs/monitoring/logging.md b/docs/monitoring/logging.md
index b548d416693..737ddb9de30 100644
--- a/docs/monitoring/logging.md
+++ b/docs/monitoring/logging.md
@@ -46,7 +46,7 @@ The configuration file either has to be specified by setting the environment pro
The `conf` directory contains a `logback.xml` file which can be modified and is used if Flink is started outside of an IDE and with the provided starting scripts.
The provided `logback.xml` has the following form:
-~~~ xml
+{% highlight xml %}
<configuration>
<appender name="file" class="ch.qos.logback.core.FileAppender">
<file>${log.file}</file>
@@ -60,13 +60,13 @@ The provided `logback.xml` has the following form:
<appender-ref ref="file"/>
</root>
</configuration>
-~~~
+{% endhighlight %}
In order to control the logging level of `org.apache.flink.runtime.jobgraph.JobGraph`, for example, one would have to add the following line to the configuration file.
-~~~ xml
+{% highlight xml %}
<logger name="org.apache.flink.runtime.jobgraph.JobGraph" level="DEBUG"/>
-~~~
+{% endhighlight %}
For further information on configuring logback see [LOGback's manual](http://logback.qos.ch/manual/configuration.html).
@@ -74,27 +74,27 @@ For further information on configuring logback see [LOGback's manual](http://log
The loggers using slf4j are created by calling
-~~~ java
+{% highlight java %}
import org.slf4j.LoggerFactory
import org.slf4j.Logger
Logger LOG = LoggerFactory.getLogger(Foobar.class)
-~~~
+{% endhighlight %}
In order to benefit most from slf4j, it is recommended to use its placeholder mechanism.
Using placeholders allows to avoid unnecessary string constructions in case that the logging level is set so high that the message would not be logged.
The syntax of placeholders is the following:
-~~~ java
+{% highlight java %}
LOG.info("This message contains {} placeholders. {}", 2, "Yippie");
-~~~
+{% endhighlight %}
Placeholders can also be used in conjunction with exceptions which shall be logged.
-~~~ java
+{% highlight java %}
catch(Exception exception){
LOG.error("An {} occurred.", "error", exception);
}
-~~~
+{% endhighlight %}
{% top %}
diff --git a/docs/monitoring/metrics.md b/docs/monitoring/metrics.md
index 1e70dd72502..55f626ed016 100644
--- a/docs/monitoring/metrics.md
+++ b/docs/monitoring/metrics.md
@@ -182,7 +182,7 @@ new class MyMapper extends RichMapFunction[String,String] {
override def open(parameters: Configuration): Unit = {
getRuntimeContext()
.getMetricGroup()
- .gauge("MyGauge", ScalaGauge[Int]( () => valueToExpose ) )
+ .gauge[Int, ScalaGauge[Int]]("MyGauge", ScalaGauge[Int]( () => valueToExpose ) )
}
override def map(value: String): String = {
@@ -426,7 +426,7 @@ class MyMapper extends RichMapFunction[Long,Long] {
Every metric is assigned an identifier and a set of key-value pairs under which the metric will be reported.
-THe identifier is based on 3 components: the user-defined name when registering the metric, an optional user-defined scope and a system-provided scope.
+The identifier is based on 3 components: a user-defined name when registering the metric, an optional user-defined scope and a system-provided scope.
For example, if `A.B` is the system scope, `C.D` the user scope and `E` the name, then the identifier for the metric will be `A.B.C.D.E`.
You can configure which delimiter to use for the identifier (default: `.`) by setting the `metrics.scope.delimiter` key in `conf/flink-conf.yaml`.
@@ -569,7 +569,7 @@ we will list more settings specific to each reporter.
Example reporter configuration that specifies multiple reporters:
-```
+{% highlight yaml %}
metrics.reporters: my_jmx_reporter,my_other_reporter
metrics.reporter.my_jmx_reporter.class: org.apache.flink.metrics.jmx.JMXReporter
@@ -579,7 +579,7 @@ metrics.reporter.my_other_reporter.class: org.apache.flink.metrics.graphite.Grap
metrics.reporter.my_other_reporter.host: 192.168.1.1
metrics.reporter.my_other_reporter.port: 10000
-```
+{% endhighlight %}
**Important:** The jar containing the reporter must be accessible when Flink is started by placing it in the /lib folder.
@@ -595,10 +595,11 @@ but not activated.
Parameters:
-- `port` - (optional) the port on which JMX listens for connections. This can also be a port range. When a
-range is specified the actual port is shown in the relevant job or task manager log. If this setting is set
-Flink will start an extra JMX connector for the given port/range. Metrics are always available on the default
-local JMX interface.
+- `port` - (optional) the port on which JMX listens for connections.
+In order to be able to run several instances of the reporter on one host (e.g. when one TaskManager is colocated with the JobManager) it is advisable to use a port range like `9250-9260`.
+When a range is specified the actual port is shown in the relevant job or task manager log.
+If this setting is set Flink will start an extra JMX connector for the given port/range.
+Metrics are always available on the default local JMX interface.
Example configuration:
@@ -699,6 +700,32 @@ Flink metric types are mapped to Prometheus metric types as follows:
All Flink metrics variables (see [List of all Variables](#list-of-all-variables)) are exported to Prometheus as labels.
+### PrometheusPushGateway (org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter)
+
+In order to use this reporter you must copy `/opt/flink-metrics-prometheus-{{site.version}}.jar` into the `/lib` folder
+of your Flink distribution.
+
+Parameters:
+
+{% include generated/prometheus_push_gateway_reporter_configuration.html %}
+
+Example configuration:
+
+{% highlight yaml %}
+
+metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
+metrics.reporter.promgateway.host: localhost
+metrics.reporter.promgateway.port: 9091
+metrics.reporter.promgateway.jobName: myJob
+metrics.reporter.promgateway.randomJobNameSuffix: true
+metrics.reporter.promgateway.deleteOnShutdown: false
+
+{% endhighlight %}
+
+The PrometheusPushGatewayReporter pushes metrics to a [Pushgateway](https://github.com/prometheus/pushgateway), which can be scraped by Prometheus.
+
+Please see the [Prometheus documentation](https://prometheus.io/docs/practices/pushing/) for use-cases.
+
### StatsD (org.apache.flink.metrics.statsd.StatsDReporter)
In order to use this reporter you must copy `/opt/flink-metrics-statsd-{{site.version}}.jar` into the `/lib` folder
@@ -1227,7 +1254,7 @@ Thus, in order to infer the metric identifier:
<td>Meter</td>
</tr>
<tr>
- <th rowspan="5"><strong>Task/Operator</strong></th>
+ <th rowspan="6"><strong>Task/Operator</strong></th>
<td>numRecordsIn</td>
<td>The total number of records this operator/task has received.</td>
<td>Counter</td>
@@ -1253,15 +1280,15 @@ Thus, in order to infer the metric identifier:
<td>Counter</td>
</tr>
<tr>
- <th rowspan="5"><strong>Operator</strong></th>
<td>currentInputWatermark</td>
<td>
- The last watermark this operator has received (in milliseconds).
- <p><strong>Note:</strong> For operators with 2 inputs this is the minimum of the last received watermarks.</p>
+ The last watermark this operator/tasks has received (in milliseconds).
+ <p><strong>Note:</strong> For operators/tasks with 2 inputs this is the minimum of the last received watermarks.</p>
</td>
<td>Gauge</td>
</tr>
<tr>
+ <th rowspan="4"><strong>Operator</strong></th>
<td>currentInput1Watermark</td>
<td>
The last watermark this operator has received in its first input (in milliseconds).
@@ -1423,7 +1450,7 @@ Request a list of available metrics:
`GET /jobmanager/metrics`
-~~~
+{% highlight json %}
[
{
"id": "metric1"
@@ -1432,13 +1459,13 @@ Request a list of available metrics:
"id": "metric2"
}
]
-~~~
+{% endhighlight %}
Request the values for specific (unaggregated) metrics:
`GET taskmanagers/ABCDE/metrics?get=metric1,metric2`
-~~~
+{% highlight json %}
[
{
"id": "metric1",
@@ -1449,13 +1476,13 @@ Request the values for specific (unaggregated) metrics:
"value": "2"
}
]
-~~~
+{% endhighlight %}
Request aggregated values for specific metrics:
`GET /taskmanagers/metrics?get=metric1,metric2`
-~~~
+{% highlight json %}
[
{
"id": "metric1",
@@ -1472,13 +1499,13 @@ Request aggregated values for specific metrics:
"sum": 16
}
]
-~~~
+{% endhighlight %}
Request specific aggregated values for specific metrics:
`GET /taskmanagers/metrics?get=metric1,metric2&agg=min,max`
-~~~
+{% highlight json %}
[
{
"id": "metric1",
@@ -1491,7 +1518,7 @@ Request specific aggregated values for specific metrics:
"max": 14,
}
]
-~~~
+{% endhighlight %}
## Dashboard integration
diff --git a/docs/monitoring/rest_api.md b/docs/monitoring/rest_api.md
index c160ad08ff4..ec92051b396 100644
--- a/docs/monitoring/rest_api.md
+++ b/docs/monitoring/rest_api.md
@@ -25,7 +25,7 @@ under the License.
Flink has a monitoring API that can be used to query status and statistics of running jobs, as well as recent completed jobs.
This monitoring API is used by Flink's own dashboard, but is designed to be used also by custom monitoring tools.
-The monitoring API is a REST-ful API that accepts HTTP GET requests and responds with JSON data.
+The monitoring API is a REST-ful API that accepts HTTP requests and responds with JSON data.
* This will be replaced by the TOC
{:toc}
@@ -33,22 +33,35 @@ The monitoring API is a REST-ful API that accepts HTTP GET requests and responds
## Overview
-The monitoring API is backed by a web server that runs as part of the *JobManager*. By default, this server listens at post `8081`, which can be configured in `flink-conf.yaml` via `jobmanager.web.port`. Note that the monitoring API web server and the web dashboard web server are currently the same and thus run together at the same port. They respond to different HTTP URLs, though.
+The monitoring API is backed by a web server that runs as part of the *Dispatcher*. By default, this server listens at post `8081`, which can be configured in `flink-conf.yaml` via `rest.port`. Note that the monitoring API web server and the web dashboard web server are currently the same and thus run together at the same port. They respond to different HTTP URLs, though.
-In the case of multiple JobManagers (for high availability), each JobManager will run its own instance of the monitoring API, which offers information about completed and running job while that JobManager was elected the cluster leader.
+In the case of multiple Dispatchers (for high availability), each Dispatcher will run its own instance of the monitoring API, which offers information about completed and running job while that Dispatcher was elected the cluster leader.
## Developing
-The REST API backend is in the `flink-runtime-web` project. The core class is `org.apache.flink.runtime.webmonitor.WebRuntimeMonitor`, which sets up the server and the request routing.
+The REST API backend is in the `flink-runtime` project. The core class is `org.apache.flink.runtime.webmonitor.WebMonitorEndpoint`, which sets up the server and the request routing.
We use *Netty* and the *Netty Router* library to handle REST requests and translate URLs. This choice was made because this combination has lightweight dependencies, and the performance of Netty HTTP is very good.
-To add new requests, one needs to add a new *request handler* class. A good example to look at is the `org.apache.flink.runtime.webmonitor.handlers.JobExceptionsHandler`. After creating the handler, the handler needs to be registered with the request router in `org.apache.flink.runtime.webmonitor.WebRuntimeMonitor`.
+To add new requests, one needs to
+* add a new `MessageHeaders` class which serves as an interface for the new request,
+* add a new `AbstractRestHandler` class which handles the request according to the added `MessageHeaders` class,
+* add the handler to `org.apache.flink.runtime.webmonitor.WebMonitorEndpoint#initializeHandlers()`.
+
+A good example is the `org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler` that uses the `org.apache.flink.runtime.rest.messages.JobExceptionsHeaders`.
## Available Requests
+### Dispatcher
+
+{% include generated/rest_dispatcher.html %}
+
+## Legacy
+
+This section is only relevant if the cluster runs in [legacy mode](../ops/config.html#mode).
+
Below is a list of available requests, with a sample JSON response. All requests are of the sample form `http://hostname:8081/jobs`, below we list only the *path* part of the URLs.
Values in angle brackets are variables, for example `http://hostname:8081/jobs/<jobid>/exceptions` will have to requested for example as `http://hostname:8081/jobs/7684be6004e4e955c2a558a9bc463f65/exceptions`.
@@ -85,7 +98,7 @@ Some information about the monitoring API and the server setup.
Sample Result:
-~~~
+{% highlight json %}
{
"refresh-interval": 3000,
"timezone-offset": 3600000,
@@ -93,7 +106,7 @@ Sample Result:
"flink-version": "{{ site.version }}",
"flink-revision": "8124545 @ 16.09.2015 @ 15:38:42 CEST"
}
-~~~
+{% endhighlight %}
**`/overview`**
@@ -101,7 +114,7 @@ Simple summary of the Flink cluster status.
Sample Result:
-~~~
+{% highlight json %}
{
"taskmanagers": 17,
"slots-total": 68,
@@ -111,17 +124,17 @@ Sample Result:
"jobs-cancelled": 1,
"jobs-failed": 0
}
-~~~
+{% endhighlight %}
### Overview of Jobs
**`/jobs/overview`**
-Jobs, grouped by status, each with a small summary of its status.
+An overview over all jobs with a small summary.
Sample Result:
-~~~
+{% highlight json %}
{
"jobs":[
{
@@ -148,7 +161,7 @@ Sample Result:
...
}]
}
-~~~
+{% endhighlight %}
### Details of a Running or Completed Job
@@ -158,7 +171,7 @@ Summary of one job, listing dataflow plan, status, timestamps of state transitio
Sample Result:
-~~~
+{% highlight json %}
{
"jid": "ab78dcdbb1db025539e30217ec54ee16",
"name": "WordCount Example",
@@ -201,7 +214,7 @@ Sample Result:
// see plan details below
}
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices`**
@@ -214,7 +227,7 @@ The user-defined execution config used by the job.
Sample Result:
-~~~
+{% highlight json %}
{
"jid": "ab78dcdbb1db025539e30217ec54ee16",
"name": "WordCount Example",
@@ -225,7 +238,7 @@ Sample Result:
"object-reuse-mode": false
}
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/exceptions`**
@@ -234,7 +247,7 @@ The `truncated` flag defines whether more exceptions occurred, but are not liste
Sample Result:
-~~~
+{% highlight json %}
{
"root-exception": "java.io.IOException: File already exists:/tmp/abzs/2\n\tat org.apache.flink.core.fs.local.LocalFileSystem. ...",
"all-exceptions": [ {
@@ -248,7 +261,7 @@ Sample Result:
} ],
"truncated":false
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/accumulators`**
@@ -256,7 +269,7 @@ The aggregated user accumulators plus job accumulators.
Sample Result:
-~~~
+{% highlight json %}
{
"job-accumulators":[],
"user-task-accumulators": [ {
@@ -270,7 +283,7 @@ Sample Result:
"value": "LongCounter 37500000"
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>`**
@@ -278,7 +291,7 @@ Information about one specific vertex, with a summary for each of its subtasks.
Sample Result:
-~~~
+{% highlight json %}
{
"id": "dceafe2df1f57a1206fcb907cb38ad97",
"name": "CHAIN DataSource -> Map -> FlatMap -> Combine(SUM(1))",
@@ -308,7 +321,7 @@ Sample Result:
}
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/subtasktimes`**
@@ -317,7 +330,7 @@ These can be used, for example, to create time-line comparisons between subtasks
Sample Result:
-~~~
+{% highlight json %}
{
"id": "dceafe2df1f57a1206fcb907cb38ad97",
"name": "CHAIN DataSource -> Map -> Combine(SUM(1))",
@@ -338,7 +351,7 @@ Sample Result:
}
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/taskmanagers`**
@@ -346,7 +359,7 @@ TaskManager statistics for one specific vertex. This is an aggregation of subtas
Sample Result:
-~~~
+{% highlight json %}
{
"id": "fe20bcc29b87cdc76589ca42114c2499",
"name": "Reduce (SUM(1), at main(WordCount.java:72)",
@@ -377,7 +390,7 @@ Sample Result:
}
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/accumulators`**
@@ -385,7 +398,7 @@ The aggregated user-defined accumulators, for a specific vertex.
Sample Result:
-~~~
+{% highlight json %}
{
"id": "dceafe2df1f57a1206fcb907cb38ad97",
"user-accumulators": [ {
@@ -394,7 +407,7 @@ Sample Result:
"name": "genwords", "type": "LongCounter", "value": "LongCounter 75000000"
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/subtasks/accumulators`**
@@ -403,7 +416,7 @@ request `/jobs/<jobid>/vertices/<vertexid>/accumulators`.
Sample Result:
-~~~
+{% highlight json %}
{
"id": "dceafe2df1f57a1206fcb907cb38ad97",
"parallelism": 2,
@@ -427,7 +440,7 @@ Sample Result:
} ]
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>`**
@@ -440,7 +453,7 @@ Summary of a specific execution attempt of a specific subtask. Multiple executio
Sample Result:
-~~~
+{% highlight json %}
{
"subtask": 0,
"status": "FINISHED",
@@ -453,7 +466,7 @@ Sample Result:
"read-bytes": 0, "write-bytes": 12684375, "read-records": 0, "write-records": 1153125
}
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>/accumulators`**
@@ -461,7 +474,7 @@ The accumulators collected for one specific subtask during one specific executio
Sample Result:
-~~~
+{% highlight json %}
{
"subtask": 0,
"attempt": 0,
@@ -474,7 +487,7 @@ Sample Result:
"name": "avglen", "type": "DoubleCounter", "value": "DoubleCounter 102.527162"
} ]
}
-~~~
+{% endhighlight %}
**`/jobs/<jobid>/plan`**
@@ -482,7 +495,7 @@ The dataflow plan of a job. The plan is also included in the job summary (`/jobs
Sample Result:
-~~~
+{% highlight json %}
{
"jid":"ab78dcdbb1db025539e30217ec54ee16",
"name":"WordCount Example",
@@ -558,7 +571,7 @@ Sample Result:
}
} ]
}
-~~~
+{% endhighlight %}
### Job Cancellation
@@ -580,54 +593,54 @@ Since savepoints can take some time to complete this operation happens asynchron
Sample Trigger Result:
-~~~
+{% highlight json %}
{
"status": "accepted",
"request-id": 1,
"location": "/jobs/:jobid/cancel-with-savepoint/in-progress/1"
}
-~~~
+{% endhighlight %}
##### Monitoring Progress
The progress of the cancellation has to be monitored by the user at
-~~~
+{% highlight json %}
/jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
-~~~
+{% endhighlight %}
The request ID is returned by the trigger result.
###### In-Progress
-~~~
+{% highlight json %}
{
"status": "in-progress",
"request-id": 1
}
-~~~
+{% endhighlight %}
###### Success
-~~~
+{% highlight json %}
{
"status": "success",
"request-id": 1,
"savepoint-path": "<savepointPath>"
}
-~~~
+{% endhighlight %}
The `savepointPath` points to the external path of the savepoint, which can be used to resume the savepoint.
###### Failed
-~~~
+{% highlight json %}
{
"status": "failed",
"request-id": 1,
"cause": "<error message>"
}
-~~~
+{% endhighlight %}
### Submitting Programs
@@ -640,11 +653,11 @@ Also make sure that the multi-part data includes the `Content-Type` of the file
The multi-part payload should start like
-```
+{% highlight plain %}
------BoundaryXXXX
Content-Disposition: form-data; name="jarfile"; filename="YourFileName.jar"
Content-Type: application/x-java-archive
-```
+{% endhighlight %}
#### Run a Program (POST)
@@ -664,18 +677,14 @@ If the call succeeds, you will get a response with the ID of the submitted job.
Request:
-~~~
+{% highlight bash %}
POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true
-~~~
+{% endhighlight %}
Response:
-~~~
+{% highlight json %}
{"jobid": "869a9868d49c679e7355700e0857af85"}
-~~~
-
-### Dispatcher
-
-{% include generated/rest_dispatcher.html %}
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/cli.md b/docs/ops/cli.md
index 45fe799ccbb..f96ccf52ea9 100644
--- a/docs/ops/cli.md
+++ b/docs/ops/cli.md
@@ -23,8 +23,8 @@ specific language governing permissions and limitations
under the License.
-->
-Flink provides a command-line interface to run programs that are packaged
-as JAR files, and control their execution. The command line interface is part
+Flink provides a Command-Line Interface (CLI) to run programs that are packaged
+as JAR files, and control their execution. The CLI is part
of any Flink setup, available in local single node setups and in
distributed setups. It is located under `<flink-home>/bin/flink`
and connects by default to the running Flink master (JobManager) that was
@@ -39,39 +39,41 @@ The command line can be used to
- submit jobs for execution,
- cancel a running job,
-- provide information about a job, and
-- list running and waiting jobs.
+- provide information about a job,
+- list running and waiting jobs,
+- trigger and dispose savepoints, and
+- modify a running job
* This will be replaced by the TOC
{:toc}
## Examples
-- Run example program with no arguments.
+- Run example program with no arguments:
./bin/flink run ./examples/batch/WordCount.jar
-- Run example program with arguments for input and result files
+- Run example program with arguments for input and result files:
./bin/flink run ./examples/batch/WordCount.jar \
--input file:///home/user/hamlet.txt --output file:///home/user/wordcount_out
-- Run example program with parallelism 16 and arguments for input and result files
+- Run example program with parallelism 16 and arguments for input and result files:
./bin/flink run -p 16 ./examples/batch/WordCount.jar \
--input file:///home/user/hamlet.txt --output file:///home/user/wordcount_out
-- Run example program with flink log output disabled
+- Run example program with flink log output disabled:
./bin/flink run -q ./examples/batch/WordCount.jar
-- Run example program in detached mode
+- Run example program in detached mode:
./bin/flink run -d ./examples/batch/WordCount.jar
- Run example program on a specific JobManager:
- ./bin/flink run -m myJMHost:6123 \
+ ./bin/flink run -m myJMHost:8081 \
./examples/batch/WordCount.jar \
--input file:///home/user/hamlet.txt --output file:///home/user/wordcount_out
@@ -104,6 +106,10 @@ The command line can be used to
./bin/flink list -r
+- List all existing jobs (including their JobIDs):
+
+ ./bin/flink list -a
+
- List running Flink jobs inside Flink YARN session:
./bin/flink list -m yarn-cluster -yid <yarnApplicationID> -r
@@ -119,9 +125,12 @@ The command line can be used to
- Stop a job (streaming jobs only):
./bin/flink stop <jobID>
+
+- Modify a running job (streaming jobs only):
+ ./bin/flink modify <jobID> -p <newParallelism>
-The difference between cancelling and stopping a (streaming) job is the following:
+**NOTE**: The difference between cancelling and stopping a (streaming) job is the following:
On a cancel call, the operators in a job immediately receive a `cancel()` method call to cancel them as
soon as possible.
@@ -148,7 +157,7 @@ This will trigger a savepoint for the job with ID `jobId`, and returns the path
Furthermore, you can optionally specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager.
-If you don't specify a target directory, you need to have [configured a default directory](#configuration) (see [Savepoints]({{site.baseurl}}/ops/state/savepoints.html#configuration)). Otherwise, triggering the savepoint will fail.
+If you don't specify a target directory, you need to have [configured a default directory]({{site.baseurl}}/ops/state/savepoints.html#configuration). Otherwise, triggering the savepoint will fail.
#### Trigger a Savepoint with YARN
@@ -165,7 +174,7 @@ Everything else is the same as described in the above **Trigger a Savepoint** se
You can atomically trigger a savepoint and cancel a job.
{% highlight bash %}
-./bin/flink cancel -s [savepointDirectory] <jobID>
+./bin/flink cancel -s [savepointDirectory] <jobID>
{% endhighlight %}
If no savepoint directory is configured, you need to configure a default savepoint directory for the Flink installation (see [Savepoints]({{site.baseurl}}/ops/state/savepoints.html#configuration)).
@@ -208,7 +217,7 @@ Otherwise, you will run into a `ClassNotFoundException`.
The command line syntax is as follows:
-~~~
+{% highlight bash %}
./flink <ACTION> [OPTIONS] [ARGUMENTS]
The following actions are available:
@@ -217,59 +226,54 @@ Action "run" compiles and runs a program.
Syntax: run [OPTIONS] <jar-file> <arguments>
"run" action options:
- -c,--class <classname> Class with the program entry
- point ("main" method or
- "getPlan()" method. Only
- needed if the JAR file does
- not specify the class in its
- manifest.
- -C,--classpath <url> Adds a URL to each user code
- classloader on all nodes in
- the cluster. The paths must
- specify a protocol (e.g.
- file://) and be accessible
- on all nodes (e.g. by means
- of a NFS share). You can use
- this option multiple times
- for specifying more than one
- URL. The protocol must be
- supported by the {@link
- java.net.URLClassLoader}.
- -d,--detached If present, runs the job in
- detached mode
- -m,--jobmanager <host:port> Address of the JobManager
- (master) to which to
- connect. Use this flag to
- connect to a different
- JobManager than the one
- specified in the
- configuration.
- -n,--allowNonRestoredState Allow non restored savepoint
- state in case an operator has
- been removed from the job.
- -p,--parallelism <parallelism> The parallelism with which
- to run the program. Optional
- flag to override the default
- value specified in the
- configuration.
- -q,--sysoutLogging If present, suppress logging
- output to standard out.
- -s,--fromSavepoint <savepointPath> Path to a savepoint to
- restore the job from (for
- example
- hdfs:///flink/savepoint-1537
- ).
- -z,--zookeeperNamespace <zookeeperNamespace> Namespace to create the
- Zookeeper sub-paths for high
- availability mode
-
+ -c,--class <classname> Class with the program entry point
+ ("main" method or "getPlan()" method.
+ Only needed if the JAR file does not
+ specify the class in its manifest.
+ -C,--classpath <url> Adds a URL to each user code
+ classloader on all nodes in the
+ cluster. The paths must specify a
+ protocol (e.g. file://) and be
+ accessible on all nodes (e.g. by means
+ of a NFS share). You can use this
+ option multiple times for specifying
+ more than one URL. The protocol must
+ be supported by the {@link
+ java.net.URLClassLoader}.
+ -d,--detached If present, runs the job in detached
+ mode
+ -n,--allowNonRestoredState Allow to skip savepoint state that
+ cannot be restored. You need to allow
+ this if you removed an operator from
+ your program that was part of the
+ program when the savepoint was
+ triggered.
+ -p,--parallelism <parallelism> The parallelism with which to run the
+ program. Optional flag to override the
+ default value specified in the
+ configuration.
+ -q,--sysoutLogging If present, suppress logging output to
+ standard out.
+ -s,--fromSavepoint <savepointPath> Path to a savepoint to restore the job
+ from (for example
+ hdfs:///flink/savepoint-1537).
Options for yarn-cluster mode:
- -yD <arg> Dynamic properties
- -yd,--yarndetached Start detached
+ -d,--detached If present, runs the job in detached
+ mode
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to
+ connect to a different JobManager than
+ the one specified in the
+ configuration.
+ -yD <property=value> use value for given property
+ -yd,--yarndetached If present, runs the job in detached
+ mode (deprecated; use non-YARN
+ specific option instead)
+ -yh,--yarnhelp Help for the Yarn session CLI.
-yid,--yarnapplicationId <arg> Attach to running YARN session
-yj,--yarnjar <arg> Path to Flink jar file
- -yjm,--yarnjobManagerMemory <arg> Memory for JobManager Container [in
- MB]
+ -yjm,--yarnjobManagerMemory <arg> Memory for JobManager Container
+ with optional unit (default: MB)
-yn,--yarncontainer <arg> Number of YARN container to allocate
(=Number of Task Managers)
-ynm,--yarnname <arg> Set a custom name for the application
@@ -281,10 +285,22 @@ Action "run" compiles and runs a program.
-yst,--yarnstreaming Start Flink in streaming mode
-yt,--yarnship <arg> Ship files in the specified directory
(t for transfer)
- -ytm,--yarntaskManagerMemory <arg> Memory per TaskManager Container [in
- MB]
+ -ytm,--yarntaskManagerMemory <arg> Memory per TaskManager Container
+ with optional unit (default: MB)
-yz,--yarnzookeeperNamespace <arg> Namespace to create the Zookeeper
sub-paths for high availability mode
+ -ynl,--yarnnodeLabel <arg> Specify YARN node label for
+ the YARN application
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
@@ -300,23 +316,30 @@ Action "info" shows the optimized execution plan of the program (JSON).
program. Optional flag to override the
default value specified in the
configuration.
- Options for yarn-cluster mode:
- -yid,--yarnapplicationId <arg> Attach to running YARN session
-
Action "list" lists running and scheduled programs.
Syntax: list [OPTIONS]
"list" action options:
- -m,--jobmanager <host:port> Address of the JobManager (master) to which
- to connect. Use this flag to connect to a
- different JobManager than the one specified
- in the configuration.
- -r,--running Show only running programs and their JobIDs
- -s,--scheduled Show only scheduled programs and their JobIDs
+ -r,--running Show only running programs and their JobIDs
+ -s,--scheduled Show only scheduled programs and their JobIDs
Options for yarn-cluster mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
-yid,--yarnapplicationId <arg> Attach to running YARN session
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
@@ -324,12 +347,23 @@ Action "stop" stops a running program (streaming jobs only).
Syntax: stop [OPTIONS] <Job ID>
"stop" action options:
- -m,--jobmanager <host:port> Address of the JobManager (master) to which
- to connect. Use this flag to connect to a
- different JobManager than the one specified
- in the configuration.
+
Options for yarn-cluster mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
-yid,--yarnapplicationId <arg> Attach to running YARN session
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
@@ -337,18 +371,27 @@ Action "cancel" cancels a running program.
Syntax: cancel [OPTIONS] <Job ID>
"cancel" action options:
- -m,--jobmanager <host:port> Address of the JobManager (master)
- to which to connect. Use this flag
- to connect to a different JobManager
- than the one specified in the
- configuration.
-s,--withSavepoint <targetDirectory> Trigger savepoint and cancel job.
The target directory is optional. If
no directory is specified, the
configured default directory
(state.savepoints.dir) is used.
Options for yarn-cluster mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
-yid,--yarnapplicationId <arg> Attach to running YARN session
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
@@ -356,14 +399,51 @@ Action "savepoint" triggers savepoints for a running job or disposes existing on
Syntax: savepoint [OPTIONS] <Job ID> [<target directory>]
"savepoint" action options:
- -d,--dispose <arg> Path of savepoint to dispose.
- -j,--jarfile <jarfile> Flink program JAR file.
- -m,--jobmanager <host:port> Address of the JobManager (master) to which
- to connect. Use this flag to connect to a
- different JobManager than the one specified
- in the configuration.
+ -d,--dispose <arg> Path of savepoint to dispose.
+ -j,--jarfile <jarfile> Flink program JAR file.
+ Options for yarn-cluster mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
+ -yid,--yarnapplicationId <arg> Attach to running YARN session
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
+
+
+
+Action "modify" modifies a running job (e.g. change of parallelism).
+
+ Syntax: modify <Job ID> [OPTIONS]
+ "modify" action options:
+ -h,--help Show the help message for the CLI
+ Frontend or the action.
+ -p,--parallelism <newParallelism> New parallelism for the specified job.
+ -v,--verbose This option is deprecated.
Options for yarn-cluster mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
-yid,--yarnapplicationId <arg> Attach to running YARN session
-~~~
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for default mode:
+ -m,--jobmanager <arg> Address of the JobManager (master) to which
+ to connect. Use this flag to connect to a
+ different JobManager than the one specified
+ in the configuration.
+ -z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
+ for high availability mode
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/config.md b/docs/ops/config.md
index d5d81f505d5..1e6be19e724 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -40,215 +40,7 @@ The configuration files for the TaskManagers can be different, Flink does not as
## Common Options
-- `env.java.home`: The path to the Java installation to use (DEFAULT: system's default Java installation, if found). Needs to be specified if the startup scripts fail to automatically resolve the java home directory. Can be specified to point to a specific java installation or version. If this option is not specified, the startup scripts also evaluate the `$JAVA_HOME` environment variable.
-
-- `env.java.opts`: Set custom JVM options. This value is respected by Flink's start scripts, both JobManager and
-TaskManager, and Flink's YARN client. This can be used to set different garbage collectors or to include remote
-debuggers into the JVMs running Flink's services. Enclosing options in double quotes delays parameter substitution
-allowing access to variables from Flink's startup scripts. Use `env.java.opts.jobmanager` and `env.java.opts.taskmanager`
-for JobManager or TaskManager-specific options, respectively.
-
-- `env.java.opts.jobmanager`: JobManager-specific JVM options. These are used in addition to the regular `env.java.opts`.
-
-- `env.java.opts.taskmanager`: TaskManager-specific JVM options. These are used in addition to the regular `env.java.opts`.
-
-- `jobmanager.rpc.address`: The external address of the JobManager, which is the master/coordinator of the distributed system (DEFAULT: localhost). **Note:** The address (host name or IP) should be accessible by all nodes including the client.
-
-- `jobmanager.rpc.port`: The port number of the JobManager (DEFAULT: 6123).
-
-- `jobmanager.heap.mb`: JVM heap size (in megabytes) for the JobManager. You may have to increase the heap size for the JobManager if you are running very large applications (with many operators), or if you are keeping a long history of them.
-
-- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible. If the cluster is exclusively running Flink, the total amount of available memory per machine minus some memory for the operating system (maybe 1-2 GB) is a good value. On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.
-
-- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: 1). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). [More about task slots](config.html#configuring-taskmanager-processing-slots).
-
-- `parallelism.default`: The default parallelism to use for programs that have no parallelism specified. (DEFAULT: 1). For setups that have no concurrent jobs running, setting this value to NumTaskManagers * NumSlotsPerTaskManager will cause the system to use all available execution resources for the program's execution. **Note**: The default parallelism can be overwritten for an entire job by calling `setParallelism(int parallelism)` on the `ExecutionEnvironment` or by passing `-p <parallelism>` to the Flink Command-line frontend. It can be overwritten for single transformations by calling `setParallelism(int
-parallelism)` on an operator. See [Parallel Execution]({{site.baseurl}}/dev/parallel.html) for more information about parallelism.
-
-- `fs.default-scheme`: The default filesystem scheme to be used, with the necessary authority to contact, e.g. the host:port of the NameNode in the case of HDFS (if needed).
-By default, this is set to `file:///` which points to the local filesystem. This means that the local
-filesystem is going to be used to search for user-specified files **without** an explicit scheme
-definition. As another example, if this is set to `hdfs://localhost:9000/`, then a user-specified file path
-without explicit scheme definition, such as `/user/USERNAME/in.txt`, is going to be transformed into
-`hdfs://localhost:9000/user/USERNAME/in.txt`. This scheme is used **ONLY** if no other scheme is specified (explicitly) in the user-provided `URI`.
-
-- `classloader.resolve-order`: Whether Flink should use a child-first `ClassLoader` when loading
-user-code classes or a parent-first `ClassLoader`. Can be one of `parent-first` or `child-first`. (default: `child-first`)
-
-- `classloader.parent-first-patterns.default`: A (semicolon-separated) list of patterns that specifies which
-classes should always be resolved through the parent `ClassLoader` first. A pattern is a simple
-prefix that is checked against the fully qualified class name. By default, this is set to
-`"java.;scala.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.slf4j;org.apache.log4j;org.apache.logging.log4j;ch.qos.logback"`.
-To extend this list beyond the default it is recommended to configure `classloader.parent-first-patterns.additional` instead of modifying this setting directly.
-
-- `classloader.parent-first-patterns.additional`: A (semicolon-separated) list of patterns that specifies which
-classes should always be resolved through the parent `ClassLoader` first. A pattern is a simple
-prefix that is checked against the fully qualified class name.
-This list is appended to `classloader.parent-first-patterns.default`.
-
-## Advanced Options
-
-### Compute
-
-- `taskmanager.compute.numa`: When enabled a TaskManager is started on each NUMA node for each worker listed in *conf/slaves* (DEFAULT: false). Note: only supported when deploying Flink as a standalone cluster.
-
-### Managed Memory
-
-By default, Flink allocates a fraction of `0.7` of the free memory (total memory configured via `taskmanager.heap.mb` minus memory used for network buffers) for its managed memory. Managed memory helps Flink to run the batch operators efficiently. It prevents `OutOfMemoryException`s because Flink knows how much memory it can use to execute operations. If Flink runs out of managed memory, it utilizes disk space. Using managed memory, some operations can be performed directly on the raw data without having to deserialize the data to convert it into Java objects. All in all, managed memory improves the robustness and speed of the system.
-
-The default fraction for managed memory can be adjusted using the `taskmanager.memory.fraction` parameter. An absolute value may be set using `taskmanager.memory.size` (overrides the fraction parameter). If desired, the managed memory may be allocated outside the JVM heap. This may improve performance in setups with large memory sizes.
-
-- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task manager reserves on-heap or off-heap (depending on `taskmanager.memory.off-heap`) for sorting, hash tables, and caching of intermediate results. If unspecified (-1), the memory manager will take a fixed ratio with respect to the size of the task manager JVM as specified by `taskmanager.memory.fraction`. (DEFAULT: -1)
-
-- `taskmanager.memory.fraction`: The relative amount of memory (with respect to `taskmanager.heap.mb`, after subtracting the amount of memory used by network buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results. For example, a value of `0.8` means that a task manager reserves 80% of its memory (on-heap or off-heap depending on `taskmanager.memory.off-heap`) for internal data buffers, leaving 20% of free memory for the task manager's heap for objects created by user-defined functions. (DEFAULT: 0.7) This parameter is only evaluated, if `taskmanager.memory.size` is not set.
-
-- `taskmanager.memory.off-heap`: If set to `true`, the task manager allocates memory which is used for sorting, hash tables, and caching of intermediate results outside of the JVM heap. For setups with larger quantities of memory, this can improve the efficiency of the operations performed on the memory (DEFAULT: false).
-
-- `taskmanager.memory.segment-size`: The size of memory buffers used by the memory manager and the network stack in bytes (DEFAULT: 32768 (= 32 KiBytes)).
-
-- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false). When `taskmanager.memory.off-heap` is set to `true`, then it is advised that this configuration is also set to `true`. If this configuration is set to `false` cleaning up of the allocated offheap memory happens only when the configured JVM parameter MaxDirectMemorySize is reached by triggering a full GC. **Note:** For streaming setups, we highly recommend to set this value to `false` as the core state backends currently do not use the managed memory.
-
-### Memory and Performance Debugging
-
-These options are useful for debugging a Flink application for memory and garbage collection related issues, such as performance and out-of-memory process kills or exceptions.
-
-- `taskmanager.debug.memory.startLogThread`: Causes the TaskManagers to periodically log memory and Garbage collection statistics. The statistics include current heap-, off-heap, and other memory pool utilization, as well as the time spent on garbage collection, by heap memory pool.
-
-- `taskmanager.debug.memory.logIntervalMs`: The interval (in milliseconds) in which the TaskManagers log the memory and garbage collection statistics. Only has an effect, if `taskmanager.debug.memory.startLogThread` is set to true.
-
-### Kerberos-based Security
-
-Flink supports Kerberos authentication for the following services:
-
-+ Hadoop Components, such as HDFS, YARN, or HBase *(version 2.6.1 and above; all other versions have critical bugs which might fail the Flink job unexpectedly)*.
-+ Kafka Connectors *(version 0.9+ and above)*.
-+ Zookeeper
-
-Configuring Flink for Kerberos security involves three aspects, explained separately in the following sub-sections.
-
-##### 1. Providing the cluster with a Kerberos credential (i.e. a keytab or a ticket via `kinit`)
-
-To provide the cluster with a Kerberos credential, Flink supports using a Kerberos keytab file or ticket caches managed by `kinit`.
-
-- `security.kerberos.login.use-ticket-cache`: Indicates whether to read from your Kerberos ticket cache (default: `true`).
-
-- `security.kerberos.login.keytab`: Absolute path to a Kerberos keytab file that contains the user credentials.
-
-- `security.kerberos.login.principal`: Kerberos principal name associated with the keytab.
-
-If both `security.kerberos.login.keytab` and `security.kerberos.login.principal` have values provided, keytabs will be used for authentication.
-It is preferable to use keytabs for long-running jobs, to avoid ticket expiration issues. If you prefer to use the ticket cache,
-talk to your administrator about increasing the Hadoop delegation token lifetime.
-
-Note that authentication using ticket caches is only supported when deploying Flink as a standalone cluster or on YARN.
-
-##### 2. Making the Kerberos credential available to components and connectors as needed
-
-For Hadoop components, Flink will automatically detect if the configured Kerberos credentials should be used when connecting to HDFS, HBase, and other Hadoop components depending on whether Hadoop security is enabled (in `core-site.xml`).
-
-For any connector or component that uses a JAAS configuration file, make the Kerberos credentials available to them by configuring JAAS login contexts for each one respectively, using the following configuration:
-
-- `security.kerberos.login.contexts`: A comma-separated list of login contexts to provide the Kerberos credentials to (for example, `Client,KafkaClient` to use the credentials for ZooKeeper authentication and for Kafka authentication).
-
-This allows enabling Kerberos authentication for different connectors or components independently. For example, you can enable Hadoop security without necessitating the use of Kerberos for ZooKeeper, or vice versa.
-
-You may also provide a static JAAS configuration file using the mechanisms described in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html), whose entries will override those produced by the above configuration option.
-
-##### 3. Configuring the component and/or connector to use Kerberos authentication
-
-Finally, be sure to configure the connector within your Flink program or component as necessary to use Kerberos authentication.
-
-Below is a list of currently first-class supported connectors or components by Flink for Kerberos authentication:
-
-- Kafka: see [here]({{site.baseurl}}/dev/connectors/kafka.html#enabling-kerberos-authentication-for-versions-above-09-only) for details on configuring the Kafka connector to use Kerberos authentication.
-
-- Zookeeper (for HA): see [here]({{site.baseurl}}/ops/jobmanager_high_availability.html#configuring-for-zookeeper-security) for details on Zookeeper security configuration to work with the Kerberos-based security configurations mentioned here.
-
-For more information on how Flink security internally setups Kerberos authentication, please see [here]({{site.baseurl}}/ops/security-kerberos.html).
-
-### Other
-
-- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round-robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: The system's tmp dir).
-
-- `taskmanager.log.path`: The config parameter defining the taskmanager log file location
-
-- `jobmanager.web.address`: Address of the JobManager's web interface (DEFAULT: anyLocalAddress()).
-
-- `jobmanager.web.port`: Port of the JobManager's web interface (DEFAULT: 8081).
-
-- `jobmanager.web.tmpdir`: This configuration parameter allows defining the Flink web directory to be used by the web interface. The web interface
-will copy its static files into the directory. Also uploaded job jars are stored in the directory if not overridden. By default, the temporary directory is used.
-
-- `jobmanager.web.upload.dir`: The config parameter defining the directory for uploading the job jars. If not specified a dynamic directory
-will be used under the directory specified by jobmanager.web.tmpdir.
-
-- `fs.overwrite-files`: Specifies whether file output writers should overwrite existing files by default. Set to *true* to overwrite by default, *false* otherwise. (DEFAULT: false)
-
-- `fs.output.always-create-directory`: File writers running with a parallelism larger than one create a directory for the output file path and put the different result files (one per parallel writer task) into that directory. If this option is set to *true*, writers with a parallelism of 1 will also create a directory and place a single result file into it. If the option is set to *false*, the writer will directly create the file directly at the output path, without creating a containing directory. (DEFAULT: false)
-
-- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. (DEFAULT: 0.1)
-
-- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: 64 MB)
-
-- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: 1 GB)
-
-- `state.backend`: The backend that will be used to store operator state checkpoints if checkpointing is enabled. Supported backends:
- - `jobmanager`: In-memory state, backup to JobManager's/ZooKeeper's memory. Should be used only for minimal state (Kafka offsets) or testing and local debugging.
- - `filesystem`: State is in-memory on the TaskManagers, and state snapshots are stored in a file system. Supported are all filesystems supported by Flink, for example HDFS, S3, ...
-
-- `state.backend.fs.checkpointdir`: Directory for storing checkpoints in a Flink supported filesystem. Note: State backend must be accessible from the JobManager, use `file://` only for local setups.
-
-- `state.backend.rocksdb.checkpointdir`: The local directory for storing RocksDB files, or a list of directories separated by the systems directory delimiter (for example ':' (colon) on Linux/Unix). (DEFAULT value is `taskmanager.tmp.dirs`)
-
-- `state.checkpoints.dir`: The target directory for meta data of [externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints).
-
-- `state.checkpoints.num-retained`: The number of completed checkpoint instances to retain. Having more than one allows recovery fallback to an earlier checkpoints if the latest checkpoint is corrupt. (Default: 1)
-
-- `high-availability.zookeeper.storageDir`: Required for HA. Directory for storing JobManager metadata; this is persisted in the state backend and only a pointer to this state is stored in ZooKeeper. Exactly like the checkpoint directory it must be accessible from the JobManager and a local filesystem should only be used for local deployments. Previously this key was named `recovery.zookeeper.storageDir`.
-
-- `blob.storage.directory`: Directory for storing blobs (such as user JARs) on the TaskManagers.
-If not set or empty, Flink will fall back to `taskmanager.tmp.dirs` and select one temp directory
-at random.
-
-- `blob.service.cleanup.interval`: Cleanup interval (in seconds) of transient blobs at server and caches as well as permanent blobs at the caches (DEFAULT: 1 hour).
-Whenever a job is not referenced at the cache anymore, we set a TTL for its permanent blob files and
-let the periodic cleanup task (executed every `blob.service.cleanup.interval` seconds) remove them
-after this TTL has passed. We do the same for transient blob files at both server and caches but
-immediately after accessing them, i.e. an put or get operation.
-This means that a blob will be retained at most <tt>2 * `blob.service.cleanup.interval`</tt> seconds after
-not being referenced anymore (permanent blobs) or their last access (transient blobs). For permanent blobs,
-this means that a recovery still has the chance to use existing files rather downloading them again.
-
-- `blob.server.port`: Port definition for the blob server (serving user JARs) on the TaskManagers. By default the port is set to 0, which means that the operating system is picking an ephemeral port. Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple JobManagers are running on the same machine.
-
-- `blob.service.ssl.enabled`: Flag to enable ssl for the blob client/server communication. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: true).
-
-- `restart-strategy`: Default [restart strategy]({{site.baseurl}}/dev/restart_strategies.html) to use in case no
-restart strategy has been specified for the job.
-The options are:
- - fixed delay strategy: `fixed-delay`.
- - failure rate strategy: `failure-rate`.
- - no restarts: `none`
-
- Default value is `none` unless checkpointing is enabled for the job in which case the default is `fixed-delay` with `Integer.MAX_VALUE` restart attempts and `10s` delay.
-
-- `restart-strategy.fixed-delay.attempts`: Number of restart attempts, used if the default restart strategy is set to "fixed-delay".
-Default value is 1, unless "fixed-delay" was activated by enabling checkpoints, in which case the default is `Integer.MAX_VALUE`.
-
-- `restart-strategy.fixed-delay.delay`: Delay between restart attempts, used if the default restart strategy is set to "fixed-delay". (default: `1 s`)
-
-- `restart-strategy.failure-rate.max-failures-per-interval`: Maximum number of restarts in given time interval before failing a job in "failure-rate" strategy.
-Default value is 1.
-
-- `restart-strategy.failure-rate.failure-rate-interval`: Time interval for measuring failure rate in "failure-rate" strategy.
-Default value is `1 minute`.
-
-- `restart-strategy.failure-rate.delay`: Delay between restart attempts, used if the default restart strategy is set to "failure-rate".
-Default value is the `akka.ask.timeout`.
-
-- `jobstore.cache-size`: The job store cache size in bytes which is used to keep completed jobs in memory (DEFAULT: `52428800` (`50` MB)).
-
-- `jobstore.expiration-time`: The time in seconds after which a completed job expires and is purged from the job store (DEFAULT: `3600`).
+{% include generated/common_section.html %}
## Full Reference
@@ -443,9 +235,9 @@ active at the same time. Since the *intra-node-parallelism* is typically the num
more than 4 repartitioning or broadcasting channels are rarely active in parallel, it frequently
boils down to
-```
+{% highlight plain %}
#slots-per-TM^2 * #TMs * 4
-```
+{% endhighlight %}
Where `#slots per TM` are the [number of slots per TaskManager](#configuring-taskmanager-processing-slots) and `#TMs` are the total number of task managers.
diff --git a/docs/ops/deployment/aws.md b/docs/ops/deployment/aws.md
index 7ef95e7399c..e62e6a3b063 100644
--- a/docs/ops/deployment/aws.md
+++ b/docs/ops/deployment/aws.md
@@ -58,9 +58,9 @@ After creating your cluster, you can [connect to the master node](http://docs.aw
1. Go the [Downloads Page]({{ site.download_url }}) and **download a binary version of Flink matching the Hadoop version** of your EMR cluster, e.g. Hadoop 2.7 for EMR releases 4.3.0, 4.4.0, or 4.5.0.
2. Extract the Flink distribution and you are ready to deploy [Flink jobs via YARN](yarn_setup.html) after **setting the Hadoop config directory**:
-```bash
+{% highlight bash %}
HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/flink run -m yarn-cluster -yn 1 examples/streaming/WordCount.jar
-```
+{% endhighlight %}
{% top %}
@@ -70,13 +70,13 @@ HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/flink run -m yarn-cluster -yn 1 examples/
You can use S3 objects like regular files by specifying paths in the following format:
-```
+{% highlight plain %}
s3://<your-bucket>/<endpoint>
-```
+{% endhighlight %}
The endpoint can either be a single file or a directory, for example:
-```java
+{% highlight java %}
// Read from S3 bucket
env.readTextFile("s3://<bucket>/<endpoint>");
@@ -85,7 +85,7 @@ stream.writeAsText("s3://<bucket>/<endpoint>");
// Use S3 as FsStatebackend
env.setStateBackend(new FsStateBackend("s3://<your-bucket>/<endpoint>"));
-```
+{% endhighlight %}
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup](../jobmanager_high_availability.html) or the [RocksDBStateBackend]({{ site.baseurl }}/ops/state/state_backends.html#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI.
@@ -101,9 +101,11 @@ implementation. Both ways are described below.
To use either `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the
`opt` directory to the `lib` directory of your Flink distribution before starting Flink, e.g.
-```
+{% highlight bash %}
cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./lib/
-```
+{% endhighlight %}
+
+Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem wrappers for URIs with the `s3://` scheme, `flink-s3-fs-hadoop` also registers for `s3a://`.
#### Configure Access Credentials
@@ -121,10 +123,10 @@ Access to S3 can be granted via your **access and secret key pair**. Please note
You need to configure both `s3.access-key` and `s3.secret-key` in Flink's `flink-conf.yaml`:
-```
+{% highlight yaml %}
s3.access-key: your-access-key
s3.secret-key: your-secret-key
-```
+{% endhighlight %}
{% top %}
@@ -149,7 +151,7 @@ This is the recommended S3 FileSystem implementation to use. It uses Amazon's SD
You need to point Flink to a valid Hadoop configuration, which contains the following properties in `core-site.xml`:
-```xml
+{% highlight xml %}
<configuration>
<property>
@@ -165,7 +167,7 @@ You need to point Flink to a valid Hadoop configuration, which contains the foll
</property>
</configuration>
-```
+{% endhighlight %}
This registers `S3AFileSystem` as the default FileSystem for URIs with the `s3a://` scheme.
@@ -175,12 +177,12 @@ This file system is limited to files up to 5GB in size and it does not work with
You need to point Flink to a valid Hadoop configuration, which contains the following property in `core-site.xml`:
-```xml
+{% highlight xml %}
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>
-```
+{% endhighlight %}
This registers `NativeS3FileSystem` as the default FileSystem for URIs with the `s3://` scheme.
@@ -192,9 +194,9 @@ You can specify the [Hadoop configuration](../config.html#hdfs) in various ways
the path of the Hadoop configuration directory, for example
- by setting the environment variable `HADOOP_CONF_DIR`, or
- by setting the `fs.hdfs.hadoopconf` configuration option in `flink-conf.yaml`:
-```
+{% highlight yaml %}
fs.hdfs.hadoopconf: /path/to/etc/hadoop
-```
+{% endhighlight %}
This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the specified directory.
@@ -222,7 +224,7 @@ Access to S3 can be granted via your **access and secret key pair**. Please note
For `S3AFileSystem` you need to configure both `fs.s3a.access.key` and `fs.s3a.secret.key` in Hadoop's `core-site.xml`:
-```xml
+{% highlight xml %}
<property>
<name>fs.s3a.access.key</name>
<value></value>
@@ -232,7 +234,7 @@ For `S3AFileSystem` you need to configure both `fs.s3a.access.key` and `fs.s3a.s
<name>fs.s3a.secret.key</name>
<value></value>
</property>
-```
+{% endhighlight %}
{% top %}
@@ -242,7 +244,7 @@ Access to S3 can be granted via your **access and secret key pair**. But this is
For `NativeS3FileSystem` you need to configure both `fs.s3.awsAccessKeyId` and `fs.s3.awsSecretAccessKey` in Hadoop's `core-site.xml`:
-```xml
+{% highlight xml %}
<property>
<name>fs.s3.awsAccessKeyId</name>
<value></value>
@@ -252,7 +254,7 @@ For `NativeS3FileSystem` you need to configure both `fs.s3.awsAccessKeyId` and `
<name>fs.s3.awsSecretAccessKey</name>
<value></value>
</property>
-```
+{% endhighlight %}
{% top %}
@@ -320,7 +322,7 @@ The following sections lists common issues when working with Flink on AWS.
If your job submission fails with an Exception message noting that `No file system found with scheme s3` this means that no FileSystem has been configured for S3. Please check out the configuration sections for our [shaded Hadoop/Presto](#shaded-hadooppresto-s3-file-systems-recommended) or [generic Hadoop](#set-s3-filesystem) file systems for details on how to configure this properly.
-```
+{% highlight plain %}
org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
Failed to submit job cd927567a81b62d7da4c18eaa91c3c39 (WordCount Example) [...]
Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error:
@@ -332,7 +334,7 @@ Caused by: java.io.IOException: No file system found with scheme s3,
at o.a.f.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450)
at o.a.f.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
at o.a.f.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
-```
+{% endhighlight %}
{% top %}
@@ -340,7 +342,7 @@ Caused by: java.io.IOException: No file system found with scheme s3,
If you see your job failing with an Exception noting that the `AWS Access Key ID and Secret Access Key must be specified as the username or password`, your access credentials have not been set up properly. Please refer to the access credential section for our [shaded Hadoop/Presto](#configure-access-credentials) or [generic Hadoop](#configure-access-credentials-1) file systems for details on how to configure this.
-```
+{% highlight plain %}
org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
Failed to submit job cd927567a81b62d7da4c18eaa91c3c39 (WordCount Example) [...]
Caused by: java.io.IOException: The given file URI (s3://<bucket>/<endpoint>) points to the
@@ -362,7 +364,7 @@ Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Acce
at o.a.h.fs.s3native.$Proxy6.initialize(Unknown Source)
at o.a.h.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:330)
at o.a.f.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:321)
-```
+{% endhighlight %}
{% top %}
@@ -370,7 +372,7 @@ Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Acce
If you see this Exception, the S3 FileSystem is not part of the class path of Flink. Please refer to [S3 FileSystem dependency section](#provide-s3-filesystem-dependency) for details on how to configure this properly.
-```
+{% highlight plain %}
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2186)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
@@ -389,7 +391,7 @@ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
... 33 more
-```
+{% endhighlight %}
{% top %}
@@ -397,16 +399,16 @@ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native
If you have configured everything properly, but get a `Bad Request` Exception **and** your S3 bucket is located in region `eu-central-1`, you might be running an S3 client, which does not support [Amazon's signature version 4](http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html).
-```
+{% highlight plain %}
[...]
Caused by: java.io.IOException: s3://<bucket-in-eu-central-1>/<endpoint> : 400 : Bad Request [...]
Caused by: org.jets3t.service.impl.rest.HttpException [...]
-```
+{% endhighlight %}
or
-```
+{% highlight plain %}
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: [...], AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: [...]
-```
+{% endhighlight %}
This should not apply to our shaded Hadoop/Presto S3 file systems but can occur for Hadoop-provided
S3 file systems. In particular, all Hadoop versions up to 2.7.2 running `NativeS3FileSystem` (which
@@ -417,9 +419,9 @@ Except for changing the bucket region, you may also be able to solve this by
[requesting signature version 4 for request authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version),
e.g. by adding this to Flink's JVM options in `flink-conf.yaml` (see
[configuration](../config.html#common-options)):
-```
+{% highlight yaml %}
env.java.opts: -Dcom.amazonaws.services.s3.enableV4
-```
+{% endhighlight %}
{% top %}
@@ -427,7 +429,7 @@ env.java.opts: -Dcom.amazonaws.services.s3.enableV4
This Exception is usually caused by skipping the local buffer directory configuration `fs.s3a.buffer.dir` for the `S3AFileSystem`. Please refer to the [S3AFileSystem configuration](#s3afilesystem-recommended) section to see how to configure the `S3AFileSystem` properly.
-```
+{% highlight plain %}
[...]
Caused by: java.lang.NullPointerException at
o.a.h.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at
@@ -442,6 +444,6 @@ o.a.h.fs.FileSystem.create(FileSystem.java:785) at
o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:404) at
o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:48) at
... 25 more
-```
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/deployment/cluster_setup.md b/docs/ops/deployment/cluster_setup.md
index c8a5d0b2845..75d0c2efff2 100644
--- a/docs/ops/deployment/cluster_setup.md
+++ b/docs/ops/deployment/cluster_setup.md
@@ -59,10 +59,10 @@ Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run pac
After downloading the latest release, copy the archive to your master node and extract it:
-~~~bash
+{% highlight bash %}
tar xzf flink-*.tgz
cd flink-*
-~~~
+{% endhighlight %}
### Configuring Flink
@@ -122,9 +122,9 @@ The following script starts a JobManager on the local node and connects via SSH
Assuming that you are on the master node and inside the Flink directory:
-~~~bash
+{% highlight bash %}
bin/start-cluster.sh
-~~~
+{% endhighlight %}
To stop Flink, there is also a `stop-cluster.sh` script.
@@ -136,15 +136,15 @@ You can add both JobManager and TaskManager instances to your running cluster wi
#### Adding a JobManager
-~~~bash
+{% highlight bash %}
bin/jobmanager.sh ((start|start-foreground) cluster)|stop|stop-all
-~~~
+{% endhighlight %}
#### Adding a TaskManager
-~~~bash
+{% highlight bash %}
bin/taskmanager.sh start|start-foreground|stop|stop-all
-~~~
+{% endhighlight %}
Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
diff --git a/docs/ops/deployment/hadoop.md b/docs/ops/deployment/hadoop.md
index cd3c096dcd9..f2f060c076f 100644
--- a/docs/ops/deployment/hadoop.md
+++ b/docs/ops/deployment/hadoop.md
@@ -40,8 +40,8 @@ running inside YARN will be started with the Hadoop classpaths, but it can
happen that the Hadoop dependencies must be in the classpath when submitting a
job to YARN. For this, it's usually enough to do a
-```
+{% highlight bash %}
export HADOOP_CLASSPATH=`hadoop classpath`
-```
+{% endhighlight %}
in the shell. Note that `hadoop` is the hadoop binary and that `classpath` is an argument that will make it print the configured Hadoop classpath.
diff --git a/docs/ops/deployment/kubernetes.md b/docs/ops/deployment/kubernetes.md
index 83c2b83a3b6..37489fe948f 100644
--- a/docs/ops/deployment/kubernetes.md
+++ b/docs/ops/deployment/kubernetes.md
@@ -42,8 +42,8 @@ Using the [resource definitions found below](#simple-kubernetes-flink-cluster-
resources), launch the cluster with the `kubectl` command:
kubectl create -f jobmanager-deployment.yaml
- kubectl create -f taskmanager-deployment.yaml
kubectl create -f jobmanager-service.yaml
+ kubectl create -f taskmanager-deployment.yaml
You can then access the Flink UI via `kubectl proxy`:
diff --git a/docs/ops/deployment/mapr_setup.md b/docs/ops/deployment/mapr_setup.md
index 19920ad9076..37b192f94b7 100644
--- a/docs/ops/deployment/mapr_setup.md
+++ b/docs/ops/deployment/mapr_setup.md
@@ -41,11 +41,11 @@ In order to run Flink on MapR, Flink needs to be built with MapR's own
Hadoop and Zookeeper distribution. Simply build Flink using Maven with
the following command from the project root directory:
-```
+{% highlight bash %}
mvn clean install -DskipTests -Pvendor-repos,mapr \
-Dhadoop.version=2.7.0-mapr-1607 \
-Dzookeeper.version=3.4.5-mapr-1604
-```
+{% endhighlight %}
The `vendor-repos` build profile adds MapR's repository to the build so that
MapR's Hadoop / Zookeeper dependencies can be fetched. The `mapr` build
@@ -67,13 +67,13 @@ The client submitting Flink jobs to MapR also needs to be prepared with the belo
Ensure that MapR's JAAS config file is picked up to avoid login failures:
-```
+{% highlight bash %}
export JVM_ARGS=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
-```
+{% endhighlight %}
Make sure that the `yarn.nodemanager.resource.cpu-vcores` property is set in `yarn-site.xml`:
-~~~xml
+{% highlight xml %}
<!-- in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-site.xml -->
<configuration>
@@ -86,28 +86,28 @@ Make sure that the `yarn.nodemanager.resource.cpu-vcores` property is set in `ya
...
</configuration>
-~~~
+{% endhighlight %}
Also remember to set the `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment
variables to the path where `yarn-site.xml` is located:
-```
+{% highlight bash %}
export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
-```
+{% endhighlight %}
Make sure that the MapR native libraries are picked up in the classpath:
-```
+{% highlight bash %}
export FLINK_CLASSPATH=/opt/mapr/lib/*
-```
+{% endhighlight %}
If you'll be starting Flink on YARN sessions with `yarn-session.sh`, the
below is also required:
-```
+{% highlight bash %}
export CC_CLASSPATH=/opt/mapr/lib/*
-```
+{% endhighlight %}
## Running Flink with a Secured MapR Cluster
@@ -125,10 +125,10 @@ Users simply need to login by using MapR's `maprlogin` authentication
utility. Users that haven't acquired MapR login credentials would not be
able to submit Flink jobs, erroring with:
-```
+{% highlight plain %}
java.lang.Exception: unable to establish the security context
Caused by: o.a.f.r.security.modules.SecurityModule$SecurityInstallException: Unable to set the Hadoop login user
Caused by: java.io.IOException: failure to login: Unable to obtain MapR credentials
-```
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/deployment/mesos.md b/docs/ops/deployment/mesos.md
index 74bae39fadb..aca6f233eb1 100644
--- a/docs/ops/deployment/mesos.md
+++ b/docs/ops/deployment/mesos.md
@@ -64,7 +64,7 @@ depends on the configuration overlay used.
The Mesos scheduler currently resides with the JobManager but will be started
independently of the JobManager in future versions (see
[FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)). The
-proposed changes will also add a Dipsatcher component which will be the central
+proposed changes will also add a Dispatcher component which will be the central
point for job submission and monitoring.
### Startup script and configuration overlays
@@ -187,7 +187,7 @@ For example:
-Dmesos.master=master.foobar.org:5050 \
-Djobmanager.heap.mb=1024 \
-Djobmanager.rpc.port=6123 \
- -Djobmanager.web.port=8081 \
+ -Drest.port=8081 \
-Dmesos.initial-tasks=10 \
-Dmesos.resourcemanager.tasks.mem=4096 \
-Dtaskmanager.heap.mb=3500 \
@@ -211,7 +211,7 @@ Here is an example configuration for Marathon:
{
"id": "flink",
- "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
+ "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
"cpus": 1.0,
"mem": 1024
}
@@ -268,6 +268,10 @@ May be set to -1 to disable this feature.
`mesos.resourcemanager.tasks.container.docker.parameters`: Custom parameters to be passed into docker run command when using the docker containerizer. Comma separated list of `key=value` pairs. `value` may contain '=' (**NO DEFAULT**)
+`mesos.resourcemanager.tasks.uris`: A comma separated list of URIs of custom artifacts to be downloaded into the sandbox of Mesos workers. (**NO DEFAULT**)
+
+`mesos.resourcemanager.tasks.container.docker.force-pull-image`: Instruct the docker containerizer to forcefully pull the image rather than reuse a cached version. (**DEFAULT**: false)
+
`mesos.resourcemanager.tasks.hostname`: Optional value to define the TaskManager's hostname. The pattern `_TASK_` is replaced by the actual id of the Mesos task. This can be used to configure the TaskManager to use Mesos DNS (e.g. `_TASK_.flink-service.mesos`) for name lookups. (**NO DEFAULT**)
`mesos.resourcemanager.tasks.bootstrap-cmd`: A command which is executed before the TaskManager is started (**NO DEFAULT**).
diff --git a/docs/ops/deployment/yarn_setup.md b/docs/ops/deployment/yarn_setup.md
index 0fb5bf6584d..9c255d27b92 100644
--- a/docs/ops/deployment/yarn_setup.md
+++ b/docs/ops/deployment/yarn_setup.md
@@ -32,14 +32,14 @@ under the License.
Start a YARN session with 4 Task Managers (each with 4 GB of Heapspace):
-~~~bash
+{% highlight bash %}
# get the hadoop2 package from the Flink download page at
# {{ site.download_url }}
curl -O <flink_hadoop2_download_url>
tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
cd flink-{{ site.version }}/
-./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096
-~~~
+./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
+{% endhighlight %}
Specify the `-s` flag for the number of processing slots per Task Manager. We recommend to set the number of slots to the number of processors per machine.
@@ -47,14 +47,14 @@ Once the session has been started, you can submit jobs to the cluster using the
### Run a Flink job on YARN
-~~~bash
+{% highlight bash %}
# get the hadoop2 package from the Flink download page at
# {{ site.download_url }}
curl -O <flink_hadoop2_download_url>
tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
cd flink-{{ site.version }}/
-./bin/flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/batch/WordCount.jar
-~~~
+./bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m ./examples/batch/WordCount.jar
+{% endhighlight %}
## Flink YARN Session
@@ -79,44 +79,44 @@ Download a Flink package for Hadoop >= 2 from the [download page]({{ site.downlo
Extract the package using:
-~~~bash
+{% highlight bash %}
tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
cd flink-{{site.version }}/
-~~~
+{% endhighlight %}
#### Start a Session
Use the following command to start a session
-~~~bash
+{% highlight bash %}
./bin/yarn-session.sh
-~~~
+{% endhighlight %}
This command will show you the following overview:
-~~~bash
+{% highlight bash %}
Usage:
Required
-n,--container <arg> Number of YARN container to allocate (=Number of Task Managers)
Optional
-D <arg> Dynamic properties
-d,--detached Start detached
- -jm,--jobManagerMemory <arg> Memory for JobManager Container [in MB]
+ -jm,--jobManagerMemory <arg> Memory for JobManager Container with optional unit (default: MB)
-nm,--name Set a custom name for the application on YARN
-q,--query Display available YARN resources (memory, cores)
-qu,--queue <arg> Specify YARN queue.
-s,--slots <arg> Number of slots per TaskManager
- -tm,--taskManagerMemory <arg> Memory per TaskManager Container [in MB]
+ -tm,--taskManagerMemory <arg> Memory per TaskManager Container with optional unit (default: MB)
-z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths for HA mode
-~~~
+{% endhighlight %}
Please note that the Client requires the `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment variable to be set to read the YARN and HDFS configuration.
**Example:** Issue the following command to allocate 10 Task Managers, with 8 GB of memory and 32 processing slots each:
-~~~bash
+{% highlight bash %}
./bin/yarn-session.sh -n 10 -tm 8192 -s 32
-~~~
+{% endhighlight %}
The system will use the configuration in `conf/flink-conf.yaml`. Please follow our [configuration guide]({{ site.baseurl }}/ops/config.html) if you want to change something.
@@ -148,25 +148,25 @@ Use the YARN utilities (`yarn application -kill <appId>`) to stop the YARN sessi
Use the following command to start a session
-~~~bash
+{% highlight bash %}
./bin/yarn-session.sh
-~~~
+{% endhighlight %}
This command will show you the following overview:
-~~~bash
+{% highlight bash %}
Usage:
Required
-id,--applicationId <yarnAppId> YARN application Id
-~~~
+{% endhighlight %}
As already mentioned, `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment variable must be set to read the YARN and HDFS configuration.
**Example:** Issue the following command to attach to running Flink YARN session `application_1463870264508_0029`:
-~~~bash
+{% highlight bash %}
./bin/yarn-session.sh -id application_1463870264508_0029
-~~~
+{% endhighlight %}
Attaching to a running session uses YARN ResourceManager to determine Job Manager RPC port.
@@ -176,15 +176,15 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Use the following command to submit a Flink program to the YARN cluster:
-~~~bash
+{% highlight bash %}
./bin/flink
-~~~
+{% endhighlight %}
Please refer to the documentation of the [command-line client]({{ site.baseurl }}/ops/cli.html).
The command will show you a help menu like this:
-~~~bash
+{% highlight bash %}
[...]
Action "run" compiles and runs a program.
@@ -202,25 +202,25 @@ Action "run" compiles and runs a program.
program. Optional flag to override the
default value specified in the
configuration
-~~~
+{% endhighlight %}
Use the *run* action to submit a job to YARN. The client is able to determine the address of the JobManager. In the rare event of a problem, you can also pass the JobManager address using the `-m` argument. The JobManager address is visible in the YARN console.
**Example**
-~~~bash
+{% highlight bash %}
wget -O LICENSE-2.0.txt http://www.apache.org/licenses/LICENSE-2.0.txt
hadoop fs -copyFromLocal LICENSE-2.0.txt hdfs:/// ...
./bin/flink run ./examples/batch/WordCount.jar \
hdfs:///..../LICENSE-2.0.txt hdfs:///.../wordcount-result.txt
-~~~
+{% endhighlight %}
If there is the following error, make sure that all TaskManagers started:
-~~~bash
+{% highlight bash %}
Exception in thread "main" org.apache.flink.compiler.CompilerException:
Available instances could not be determined from job manager: Connection timed out.
-~~~
+{% endhighlight %}
You can check the number of TaskManagers in the JobManager web interface. The address of this interface is printed in the YARN session console.
@@ -235,9 +235,9 @@ Please note that the client then expects the `-yn` value to be set (number of Ta
***Example:***
-~~~bash
+{% highlight bash %}
./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar
-~~~
+{% endhighlight %}
The command line options of the YARN session are also available with the `./bin/flink` tool. They are prefixed with a `y` or `yarn` (for the long argument options).
@@ -275,9 +275,9 @@ In cases where the Flink YARN session fails during the deployment itself, users
To enable it, users have to set the `yarn.log-aggregation-enable` property to `true` in the `yarn-site.xml` file.
Once that is enabled, users can use the following command to retrieve all log files of a (failed) YARN session.
-~~~
+{% highlight bash %}
yarn logs -applicationId <application ID>
-~~~
+{% endhighlight %}
Note that it takes a few seconds after the session has finished until the logs show up.
diff --git a/docs/ops/filesystems.md b/docs/ops/filesystems.md
index 75ddf6d156a..dab38176493 100644
--- a/docs/ops/filesystems.md
+++ b/docs/ops/filesystems.md
@@ -73,18 +73,18 @@ That way, Flink seamlessly supports all of Hadoop file systems, and all Hadoop-c
## Common File System configurations
-The following configuration settings exist across different file systems
+The following configuration settings exist across different file systems.
#### Default File System
If paths to files do not explicitly specify a file system scheme (and authority), a default scheme (and authority) will be used.
-~~~
+{% highlight yaml %}
fs.default-scheme: <default-fs>
-~~~
+{% endhighlight %}
-For example, if the default file system configured as `fs.default-scheme: hdfs://localhost:9000/`, then a a file path of
-`/user/hugo/in.txt'` is interpreted as `hdfs://localhost:9000/user/hugo/in.txt'`
+For example, if the default file system configured as `fs.default-scheme: hdfs://localhost:9000/`, then a file path of
+`/user/hugo/in.txt` is interpreted as `hdfs://localhost:9000/user/hugo/in.txt`.
#### Connection limiting
@@ -96,13 +96,13 @@ For example, very small HDFS clusters with few RPC handlers can sometimes be ove
To limit a specific file system's connections, add the following entries to the Flink configuration. The file system to be limited is identified by
its scheme.
-~~~
+{% highlight yaml %}
fs.<scheme>.limit.total: (number, 0/-1 mean no limit)
fs.<scheme>.limit.input: (number, 0/-1 mean no limit)
fs.<scheme>.limit.output: (number, 0/-1 mean no limit)
fs.<scheme>.limit.timeout: (milliseconds, 0 means infinite)
fs.<scheme>.limit.stream-timeout: (milliseconds, 0 means infinite)
-~~~
+{% endhighlight %}
You can limit the number if input/output connections (streams) separately (`fs.<scheme>.limit.input` and `fs.<scheme>.limit.output`), as well as impose a limit on
the total number of concurrent streams (`fs.<scheme>.limit.total`). If the file system tries to open more streams, the operation will block until some streams are closed.
@@ -112,7 +112,7 @@ To prevent inactive streams from taking up the complete pool (preventing new con
`fs.<scheme>.limit.stream-timeout`. If a stream does not read/write any bytes for at least that amount of time, it is forcibly closed.
These limits are enforced per TaskManager, so each TaskManager in a Flink application or cluster will open up to that number of connections.
-In addition, the The limit are also enforced only per FileSystem instance. Because File Systems are created per scheme and authority, different
+In addition, the limits are also only enforced per FileSystem instance. Because File Systems are created per scheme and authority, different
authorities will have their own connection pool. For example `hdfs://myhdfs:50010/` and `hdfs://anotherhdfs:4399/` will have separate pools.
diff --git a/docs/ops/jobmanager_high_availability.md b/docs/ops/jobmanager_high_availability.md
index a99455849c4..320f9d62e50 100644
--- a/docs/ops/jobmanager_high_availability.md
+++ b/docs/ops/jobmanager_high_availability.md
@@ -93,7 +93,7 @@ In order to start an HA-cluster add the following configuration keys to `conf/fl
- **Storage directory** (required): JobManager metadata is persisted in the file system *storageDir* and only a pointer to this state is stored in ZooKeeper.
<pre>
-high-availability.zookeeper.storageDir: hdfs:///flink/recovery
+high-availability.storageDir: hdfs:///flink/recovery
</pre>
The `storageDir` stores all metadata needed to recover a JobManager failure.
@@ -109,7 +109,7 @@ high-availability: zookeeper
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /cluster_one # important: customize per cluster
-high-availability.zookeeper.storageDir: hdfs:///flink/recovery</pre>
+high-availability.storageDir: hdfs:///flink/recovery</pre>
2. **Configure masters** in `conf/masters`:
@@ -191,7 +191,7 @@ This means that the application can be restarted 9 times for failed attempts bef
<pre>
high-availability: zookeeper
high-availability.zookeeper.quorum: localhost:2181
-high-availability.zookeeper.storageDir: hdfs:///flink/recovery
+high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 10</pre>
diff --git a/docs/ops/production_ready.md b/docs/ops/production_ready.md
index 0d11b8a1866..08fef2bb10d 100644
--- a/docs/ops/production_ready.md
+++ b/docs/ops/production_ready.md
@@ -87,4 +87,14 @@ you are sure that your state will never exceed main memory and blocking the stre
you **could consider** to not use the RocksDB backends. However, at this point, we **strongly recommend** using RocksDB
for production.
+### Config JobManager High Availability(HA)
+
+The JobManager coordinates every Flink deployment. It is responsible for both *scheduling* and *resource management*.
+
+By default, there is a single JobManager instance per Flink cluster. This creates a *single point of failure* (SPOF):
+if the JobManager crashes, no new programs can be submitted and running programs fail.
+
+With JobManager High Availability, you can recover from JobManager failures and thereby eliminate the *SPOF*.
+We **strongly recommend** you configure [high availability]({{ site.baseurl }}/ops/jobmanager_high_availability.html) for production.
+
{% top %}
diff --git a/docs/ops/security-kerberos.md b/docs/ops/security-kerberos.md
index 5589057a327..0a831b6baec 100644
--- a/docs/ops/security-kerberos.md
+++ b/docs/ops/security-kerberos.md
@@ -53,7 +53,7 @@ Flink provides first-class support for Kerberos authentication only. The follow
Note that it is possible to enable the use of Kerberos independently for each service or connector. For example, the user may enable
Hadoop security without necessitating the use of Kerberos for ZooKeeper, or vice versa. The shared element is the configuration of
-Kerbreros credentials, which is then explicitly used by each component.
+Kerberos credentials, which is then explicitly used by each component.
The internal architecture is based on security modules (implementing `org.apache.flink.runtime.security.modules.SecurityModule`) which
are installed at startup. The following sections describes each security module.
diff --git a/docs/ops/security-ssl.md b/docs/ops/security-ssl.md
index 961af892f58..1a3c3810250 100644
--- a/docs/ops/security-ssl.md
+++ b/docs/ops/security-ssl.md
@@ -22,124 +22,137 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to enable SSL for the network communication between different flink components.
+This page provides instructions on how to enable SSL for the network communication between different Flink components.
## SSL Configuration
-SSL can be enabled for all network communication between flink components. SSL keystores and truststore has to be deployed on each flink node and configured (conf/flink-conf.yaml) using keys in the security.ssl.* namespace (Please see the [configuration page](config.html) for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when security.ssl.enabled is set to true.
+SSL can be enabled for all network communication between Flink components. SSL keystores and truststore has to be deployed on each Flink node and configured (conf/flink-conf.yaml) using keys in the security.ssl.* namespace (Please see the [configuration page](config.html) for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when security.ssl.enabled is set to true.
* **taskmanager.data.ssl.enabled**: SSL flag for data communication between task managers
* **blob.service.ssl.enabled**: SSL flag for blob service client/server communication
-* **akka.ssl.enabled**: SSL flag for the akka based control connection between the flink client, jobmanager and taskmanager
+* **akka.ssl.enabled**: SSL flag for akka based control connection between the Flink client, jobmanager and taskmanager
* **jobmanager.web.ssl.enabled**: Flag to enable https access to the jobmanager's web frontend
+**IMPORTANT**
+
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends to use a specific set of cipher suites for strong security.
+Because these cipher suites were not available on many setups out of the box, Flink's default value is set to a slightly
+weaker but more compatible cipher suite.
+We recommend that SSL setups update to the stronger cipher suites, if possible, by adding the below entry to the Flink configuration:
+
+{% highlight yaml %}
+security.ssl.algorithms: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
+{% endhighlight %}
+
+If these suites are not supported on your setup, you will see that Flink processes will not be able to connect to each other.
+
## Deploying Keystores and Truststores
-You need to have a Java Keystore generated and copied to each node in the flink cluster. The common name or subject alternative names in the certificate should match the node's hostname and IP address. Keystores and truststores can be generated using the keytool utility (https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html). All flink components should have read access to the keystore and truststore files.
+You need to have a Java Keystore generated and copied to each node in the Flink cluster. The common name or subject alternative names in the certificate should match the node's hostname and IP address. Keystores and truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html). All Flink components should have read access to the keystore and truststore files.
-### Example: Creating self signed CA and keystores for a 2 node cluster
+### Example: Creating self signed CA and keystores for a two-node cluster
-Execute the following keytool commands to create a truststore with a self signed CA
+Execute the following keytool commands to create a truststore with a self signed CA.
-~~~
+{% highlight bash %}
keytool -genkeypair -alias ca -keystore ca.keystore -dname "CN=Sample CA" -storepass password -keypass password -keyalg RSA -ext bc=ca:true
keytool -keystore ca.keystore -storepass password -alias ca -exportcert > ca.cer
keytool -importcert -keystore ca.truststore -alias ca -storepass password -noprompt -file ca.cer
-~~~
+{% endhighlight %}
Now create keystores for each node with certificates signed by the above CA. Let node1.company.org and node2.company.org be the hostnames with IPs 192.168.1.1 and 192.168.1.2 respectively
#### Node 1
-~~~
+{% highlight bash %}
keytool -genkeypair -alias node1 -keystore node1.keystore -dname "CN=node1.company.org" -ext SAN=dns:node1.company.org,ip:192.168.1.1 -storepass password -keypass password -keyalg RSA
keytool -certreq -keystore node1.keystore -storepass password -alias node1 -file node1.csr
keytool -gencert -keystore ca.keystore -storepass password -alias ca -ext SAN=dns:node1.company.org,ip:192.168.1.1 -infile node1.csr -outfile node1.cer
keytool -importcert -keystore node1.keystore -storepass password -file ca.cer -alias ca -noprompt
keytool -importcert -keystore node1.keystore -storepass password -file node1.cer -alias node1 -noprompt
-~~~
+{% endhighlight %}
#### Node 2
-~~~
+{% highlight bash %}
keytool -genkeypair -alias node2 -keystore node2.keystore -dname "CN=node2.company.org" -ext SAN=dns:node2.company.org,ip:192.168.1.2 -storepass password -keypass password -keyalg RSA
keytool -certreq -keystore node2.keystore -storepass password -alias node2 -file node2.csr
keytool -gencert -keystore ca.keystore -storepass password -alias ca -ext SAN=dns:node2.company.org,ip:192.168.1.2 -infile node2.csr -outfile node2.cer
keytool -importcert -keystore node2.keystore -storepass password -file ca.cer -alias ca -noprompt
keytool -importcert -keystore node2.keystore -storepass password -file node2.cer -alias node2 -noprompt
-~~~
+{% endhighlight %}
## Standalone Deployment
Configure each node in the standalone cluster to pick up the keystore and truststore files present in the local file system.
-### Example: 2 node cluster
+### Example: Two-node cluster
-* Generate 2 keystores, one for each node, and copy them to the filesystem on the respective node. Also copy the pulic key of the CA (which was used to sign the certificates in the keystore) as a Java truststore on both the nodes
-* Configure conf/flink-conf.yaml to pick up these files
+* Generate two keystores, one for each node, and copy them to the filesystem on the respective node. Also copy the public key of the CA (which was used to sign the certificates in the keystore) as a Java truststore on both the nodes.
+* Configure conf/flink-conf.yaml to pick up these files.
#### Node 1
-~~~
+{% highlight yaml %}
security.ssl.enabled: true
security.ssl.keystore: /usr/local/node1.keystore
-security.ssl.keystore-password: abc123
-security.ssl.key-password: abc123
+security.ssl.keystore-password: password
+security.ssl.key-password: password
security.ssl.truststore: /usr/local/ca.truststore
-security.ssl.truststore-password: abc123
-~~~
+security.ssl.truststore-password: password
+{% endhighlight %}
#### Node 2
-~~~
+{% highlight yaml %}
security.ssl.enabled: true
security.ssl.keystore: /usr/local/node2.keystore
-security.ssl.keystore-password: abc123
-security.ssl.key-password: abc123
+security.ssl.keystore-password: password
+security.ssl.key-password: password
security.ssl.truststore: /usr/local/ca.truststore
-security.ssl.truststore-password: abc123
-~~~
+security.ssl.truststore-password: password
+{% endhighlight %}
-* Restart the flink components to enable SSL for all of flink's internal communication
-* Verify by accessing the jobmanager's UI using https url. The task manager's path in the UI should show akka.ssl.tcp:// as the protocol
-* The blob server and task manager's data communication can be verified from the log files
+* Restart the Flink components to enable SSL for all of Flink's internal communication
+* Verify by accessing the jobmanager's UI using https url. The taskmanager's path in the UI should show akka.ssl.tcp:// as the protocol
+* The blob server and taskmanager's data communication can be verified from the log files
## YARN Deployment
-The keystores and truststore can be deployed in a YARN setup in multiple ways depending on the cluster setup. Following are 2 ways to achieve this
+The keystores and truststore can be deployed in a YARN setup in multiple ways depending on the cluster setup. Following are two ways to achieve this.
### 1. Deploy keystores before starting the YARN session
-The keystores and truststore should be generated and deployed on all nodes in the YARN setup where flink components can potentially be executed. The same flink config file from the flink YARN client is used for all the flink components running in the YARN cluster. Therefore we need to ensure the keystore is deployed and accessible using the same filepath in all the YARN nodes.
+The keystores and truststore should be generated and deployed on all nodes in the YARN setup where Flink components can potentially be executed. The same Flink config file from the Flink YARN client is used for all the Flink components running in the YARN cluster. Therefore we need to ensure the keystore is deployed and accessible using the same filepath in all the YARN nodes.
#### Example config
-~~~
+{% highlight yaml %}
security.ssl.enabled: true
security.ssl.keystore: /usr/local/node.keystore
-security.ssl.keystore-password: abc123
-security.ssl.key-password: abc123
+security.ssl.keystore-password: password
+security.ssl.key-password: password
security.ssl.truststore: /usr/local/ca.truststore
-security.ssl.truststore-password: abc123
-~~~
+security.ssl.truststore-password: password
+{% endhighlight %}
Now you can start the YARN session from the CLI like you would normally do.
-### 2. Use YARN cli to deploy the keystores and truststore
-We can use the YARN client's ship files option (-yt) to distribute the keystores and truststore. Since the same keystore will be deployed at all nodes, we need to ensure a single certificate in the keystore can be served for all nodes. This can be done by either using the Subject Alternative Name(SAN) extension in the certificate and setting it to cover all nodes (hostname and ip addresses) in the cluster or by using wildcard subdomain names (if the cluster is setup accordingly).
+### 2. Use YARN CLI to deploy the keystores and truststore
+We can use the YARN client's ship files option (-yt) to distribute the keystores and truststore. Since the same keystore will be deployed at all nodes, we need to ensure a single certificate in the keystore can be served for all nodes. This can be done by either using the Subject Alternative Name (SAN) extension in the certificate and setting it to cover all nodes (hostname and ip addresses) in the cluster or by using wildcard subdomain names (if the cluster is setup accordingly).
#### Example
* Supply the following parameters to the keytool command when generating the keystore: -ext SAN=dns:node1.company.org,ip:192.168.1.1,dns:node2.company.org,ip:192.168.1.2
-* Copy the keystore and the CA's truststore into a local directory (at the cli's working directory), say deploy-keys/
+* Copy the keystore and the CA's truststore into a local directory (at the CLI's working directory), say deploy-keys/
* Update the configuration to pick up the files from a relative path
-~~~
+{% highlight yaml %}
security.ssl.enabled: true
security.ssl.keystore: deploy-keys/node.keystore
security.ssl.keystore-password: password
security.ssl.key-password: password
security.ssl.truststore: deploy-keys/ca.truststore
security.ssl.truststore-password: password
-~~~
+{% endhighlight %}
* Start the YARN session using the -yt parameter
-~~~
+{% highlight bash %}
flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar
-~~~
+{% endhighlight %}
-When deployed using YARN, flink's web dashboard is accessible through YARN proxy's Tracking URL. To ensure that the YARN proxy is able to access flink's https url you need to configure YARN proxy to accept flink's SSL certificates. Add the custom CA certificate into Java's default truststore on the YARN Proxy node.
+When deployed using YARN, Flink's web dashboard is accessible through YARN proxy's Tracking URL. To ensure that the YARN proxy is able to access Flink's https url you need to configure YARN proxy to accept Flink's SSL certificates. Add the custom CA certificate into Java's default truststore on the YARN Proxy node.
{% top %}
diff --git a/docs/ops/state/checkpoints.md b/docs/ops/state/checkpoints.md
index d2df4e6cf33..c68bb0f1ed5 100644
--- a/docs/ops/state/checkpoints.md
+++ b/docs/ops/state/checkpoints.md
@@ -35,68 +35,63 @@ the same semantics as a failure-free execution.
See [Checkpointing]({{ site.baseurl }}/dev/stream/state/checkpointing.html) for how to enable and
configure checkpoints for your program.
-## Externalized Checkpoints
+## Retained Checkpoints
-Checkpoints are by default not persisted externally and are only used to
-resume a job from failures. They are deleted when a program is cancelled.
-You can, however, configure periodic checkpoints to be persisted externally
-similarly to [savepoints](savepoints.html). These *externalized checkpoints*
-write their meta data out to persistent storage and are *not* automatically
-cleaned up when the job fails. This way, you will have a checkpoint around
-to resume from if your job fails.
+Checkpoints are by default not retained and are only used to resume a
+job from failures. They are deleted when a program is cancelled.
+You can, however, configure periodic checkpoints to be retained.
+Depending on the configuration these *retained* checkpoints are *not*
+automatically cleaned up when the job fails or is canceled.
+This way, you will have a checkpoint around to resume from if your job fails.
-```java
+{% highlight java %}
CheckpointConfig config = env.getCheckpointConfig();
config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
-```
+{% endhighlight %}
-The `ExternalizedCheckpointCleanup` mode configures what happens with externalized checkpoints when you cancel the job:
+The `ExternalizedCheckpointCleanup` mode configures what happens with checkpoints when you cancel the job:
-- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the externalized checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
+- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
-- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the externalized checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
+- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
### Directory Structure
-Similarly to [savepoints](savepoints.html), an externalized checkpoint consists
-of a meta data file and, depending on the state back-end, some additional data
-files. The **target directory** for the externalized checkpoint's meta data is
-determined from the configuration key `state.checkpoints.dir` which, currently,
-can only be set via the configuration files.
+Similarly to [savepoints](savepoints.html), a checkpoint consists
+of a meta data file and, depending on the state backend, some additional data
+files. The meta data file and data files are stored in the directory that is
+configured via `state.checkpoints.dir` in the configuration files,
+and also can be specified for per job in the code.
-```
-state.checkpoints.dir: hdfs:///checkpoints/
-```
+#### Configure globally via configuration files
-This directory will then contain the checkpoint meta data required to restore
-the checkpoint. For the `MemoryStateBackend`, this meta data file will be
-self-contained and no further files are needed.
+{% highlight yaml %}
+state.checkpoints.dir: hdfs:///checkpoints/
+{% endhighlight %}
-`FsStateBackend` and `RocksDBStateBackend` write separate data files
-and only write the paths to these files into the meta data file. These data
-files are stored at the path given to the state back-end during construction.
+#### Configure for per job when constructing the state backend
-```java
+{% highlight java %}
env.setStateBackend(new RocksDBStateBackend("hdfs:///checkpoints-data/");
-```
+{% endhighlight %}
### Difference to Savepoints
-Externalized checkpoints have a few differences from [savepoints](savepoints.html). They
+Checkpoints have a few differences from [savepoints](savepoints.html). They
- use a state backend specific (low-level) data format, may be incremental.
- do not support Flink specific features like rescaling.
-### Resuming from an externalized checkpoint
+### Resuming from a retained checkpoint
-A job may be resumed from an externalized checkpoint just as from a savepoint
+A job may be resumed from a checkpoint just as from a savepoint
by using the checkpoint's meta data file instead (see the
[savepoint restore guide](../cli.html#restore-a-savepoint)). Note that if the
meta data file is not self-contained, the jobmanager needs to have access to
the data files it refers to (see [Directory Structure](#directory-structure)
above).
-```sh
+{% highlight shell %}
$ bin/flink run -s :checkpointMetaDataPath [:runArgs]
-```
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/state/large_state_tuning.md b/docs/ops/state/large_state_tuning.md
index 635993849f7..6df551f32da 100644
--- a/docs/ops/state/large_state_tuning.md
+++ b/docs/ops/state/large_state_tuning.md
@@ -298,10 +298,7 @@ that the task-local state is an in-memory consisting of heap objects, and not st
### Configuring task-local recovery
Task-local recovery is *deactivated by default* and can be activated through Flink's configuration with the key `state.backend.local-recovery` as specified
-in `CheckpointingOptions.LOCAL_RECOVERY`. Users have currently two choices:
-
-- `DISABLED`: Local recovery is disabled (default).
-- `ENABLE_FILE_BASED`: Local recovery is activated, based on writing a secondary copies of the task state on local disk.
+in `CheckpointingOptions.LOCAL_RECOVERY`. The value for this setting can either be *true* to enable or *false* (default) to disable local recovery.
### Details on task-local recovery for different state backends
diff --git a/docs/ops/state/savepoints.md b/docs/ops/state/savepoints.md
index 1a714b3a5d4..6dd5154c5e6 100644
--- a/docs/ops/state/savepoints.md
+++ b/docs/ops/state/savepoints.md
@@ -59,12 +59,12 @@ If you don't specify the IDs manually they will be generated automatically. You
You can think of a savepoint as holding a map of `Operator ID -> State` for each stateful operator:
-```
+{% highlight plain %}
Operator ID | State
------------+------------------------
source-id | State of StatefulSource
mapper-id | State of StatefulMapper
-```
+{% endhighlight %}
In the above example, the print sink is stateless and hence not part of the savepoint state. By default, we try to map each entry of the savepoint back to the new program.
@@ -84,7 +84,7 @@ When triggering a savepoint, a new savepoint directory is created where the data
For example with a `FsStateBackend` or `RocksDBStateBackend`:
-```sh
+{% highlight shell %}
# Savepoint target directory
/savepoints/
@@ -96,7 +96,7 @@ For example with a `FsStateBackend` or `RocksDBStateBackend`:
# Savepoint state
/savepoints/savepoint-:shortjobid-:savepointid/...
-```
+{% endhighlight %}
<div class="alert alert-info">
<strong>Note:</strong>
@@ -108,33 +108,33 @@ Note that if you use the `MemoryStateBackend`, metadata *and* savepoint state wi
#### Trigger a Savepoint
-```sh
+{% highlight shell %}
$ bin/flink savepoint :jobId [:targetDirectory]
-```
+{% endhighlight %}
This will trigger a savepoint for the job with ID `:jobId`, and returns the path of the created savepoint. You need this path to restore and dispose savepoints.
#### Trigger a Savepoint with YARN
-```sh
+{% highlight shell %}
$ bin/flink savepoint :jobId [:targetDirectory] -yid :yarnAppId
-```
+{% endhighlight %}
This will trigger a savepoint for the job with ID `:jobId` and YARN application ID `:yarnAppId`, and returns the path of the created savepoint.
#### Cancel Job with Savepoint
-```sh
+{% highlight shell %}
$ bin/flink cancel -s [:targetDirectory] :jobId
-```
+{% endhighlight %}
This will atomically trigger a savepoint for the job with ID `:jobid` and cancel the job. Furthermore, you can specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager(s) and TaskManager(s).
### Resuming from Savepoints
-```sh
+{% highlight shell %}
$ bin/flink run -s :savepointPath [:runArgs]
-```
+{% endhighlight %}
This submits a job and specifies a savepoint to resume from. You may give a path to either the savepoint's directory or the `_metadata` file.
@@ -142,15 +142,15 @@ This submits a job and specifies a savepoint to resume from. You may give a path
By default the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option:
-```sh
+{% highlight shell %}
$ bin/flink run -s :savepointPath -n [:runArgs]
-```
+{% endhighlight %}
### Disposing Savepoints
-```sh
+{% highlight shell %}
$ bin/flink savepoint -d :savepointPath
-```
+{% endhighlight %}
This disposes the savepoint stored in `:savepointPath`.
@@ -160,10 +160,10 @@ Note that it is possible to also manually delete a savepoint via regular file sy
You can configure a default savepoint target directory via the `state.savepoints.dir` key. When triggering savepoints, this directory will be used to store the savepoint. You can overwrite the default by specifying a custom target directory with the trigger commands (see the [`:targetDirectory` argument](#trigger-a-savepoint)).
-```sh
+{% highlight yaml %}
# Default savepoint target directory
state.savepoints.dir: hdfs:///flink/savepoints
-```
+{% endhighlight %}
If you neither configure a default nor specify a custom target directory, triggering the savepoint will fail.
@@ -189,9 +189,9 @@ By default, a savepoint restore will try to match all state back to the restored
You can allow non restored state by setting the `--allowNonRestoredState` (short: `-n`) with the run command:
-```sh
+{% highlight shell %}
$ bin/flink run -s :savepointPath -n [:runArgs]
-```
+{% endhighlight %}
### What happens if I reorder stateful operators in my job?
diff --git a/docs/ops/state/state_backends.md b/docs/ops/state/state_backends.md
index 3410623d289..3d4ce586759 100644
--- a/docs/ops/state/state_backends.md
+++ b/docs/ops/state/state_backends.md
@@ -152,11 +152,12 @@ Possible values for the config entry are *jobmanager* (MemoryStateBackend), *fil
name of the class that implements the state backend factory [FsStateBackendFactory](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackendFactory.java),
such as `org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory` for RocksDBStateBackend.
-In the case where the default state backend is set to *filesystem*, the entry `state.backend.fs.checkpointdir` defines the directory where the checkpoint data will be stored.
+The `state.checkpoints.dir` option defines the directory to which all backends write checkpoint data and meta data files.
+You can find more details about the checkpoint directory structure [here](checkpoints.html#directory-structure).
A sample section in the configuration file could look as follows:
-~~~
+{% highlight yaml %}
# The backend that will be used to store operator state checkpoints
state.backend: filesystem
@@ -164,7 +165,7 @@ state.backend: filesystem
# Directory for storing checkpoints
-state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints
-~~~
+state.checkpoints.dir: hdfs://namenode:40010/flink/checkpoints
+{% endhighlight %}
{% top %}
diff --git a/docs/ops/upgrading.md b/docs/ops/upgrading.md
index f5c8ee02e09..475dd402719 100644
--- a/docs/ops/upgrading.md
+++ b/docs/ops/upgrading.md
@@ -36,21 +36,21 @@ The line of action for upgrading a streaming application or migrating an applica
There are two ways of taking a savepoint from a running streaming application.
* Taking a savepoint and continue processing.
-```
+{% highlight bash %}
> ./bin/flink savepoint <jobID> [pathToSavepoint]
-```
+{% endhighlight %}
It is recommended to periodically take savepoints in order to be able to restart an application from a previous point in time.
* Taking a savepoint and stopping the application as a single action.
-```
+{% highlight bash %}
> ./bin/flink cancel -s [pathToSavepoint] <jobID>
-```
+{% endhighlight %}
This means that the application is canceled immediately after the savepoint completed, i.e., no other checkpoints are taken after the savepoint.
Given a savepoint taken from an application, the same or a compatible application (see [Application State Compatibility](#application-state-compatibility) section below) can be started from that savepoint. Starting an application from a savepoint means that the state of its operators is initialized with the operator state persisted in the savepoint. This is done by starting an application using a savepoint.
-```
+{% highlight bash %}
> ./bin/flink run -d -s [pathToSavepoint] ~/application.jar
-```
+{% endhighlight %}
The operators of the started application are initialized with the operator state of the original application (i.e., the application the savepoint was taken from) at the time when the savepoint was taken. The started application continues processing from exactly this point on.
@@ -66,10 +66,10 @@ In this section, we discuss how applications can be modified to remain state com
When an application is restarted from a savepoint, Flink matches the operator state stored in the savepoint to stateful operators of the started application. The matching is done based on operator IDs, which are also stored in the savepoint. Each operator has a default ID that is derived from the operator's position in the application's operator topology. Hence, an unmodified application can always be restarted from one of its own savepoints. However, the default IDs of operators are likely to change if an application is modified. Therefore, modified applications can only be started from a savepoint if the operator IDs have been explicitly specified. Assigning IDs to operators is very simple and done using the `uid(String)` method as follows:
-```
+{% highlight scala%}
val mappedEvents: DataStream[(Int, Long)] = events
- .map(new MyStatefulMapFunc()).uid(“mapper-1”)
-```
+ .map(new MyStatefulMapFunc()).uid("mapper-1")
+{% endhighlight %}
**Note:** Since the operator IDs stored in a savepoint and IDs of operators in the application to start must be equal, it is highly recommended to assign unique IDs to all operators of an application that might be upgraded in the future. This advice applies to all operators, i.e., operators with and without explicitly declared operator state, because some operators have internal state that is not visible to the user. Upgrading an application without assigned operator IDs is significantly more difficult and may only be possible via a low-level workaround using the `setUidHash()` method.
@@ -107,7 +107,7 @@ When upgrading an application by changing its topology, a few things need to be
* **Adding a stateful operator:** The state of the operator will be initialized with the default state unless it takes over the state of another operator.
* **Removing a stateful operator:** The state of the removed operator is lost unless another operator takes it over. When starting the upgraded application, you have to explicitly agree to discard the state.
* **Changing of input and output types of operators:** When adding a new operator before or behind an operator with internal state, you have to ensure that the input or output type of the stateful operator is not modified to preserve the data type of the internal operator state (see above for details).
-* **Changing operator chaining:** Operators can be chained together for improved performance. When restoring from a savepoint taken since 1.3.x it is possible to modify chains while preserving state consistency. It is possible a break the chain such that a stateful operator is moved out of the chain. It is also possible to append or inject a new or existing stateful operator into a chain, or to modify the operator order within a chain. However, when upgrading a savepoint to 1.3.x it is paramount that the topology did not change in regards to chaining. All operators that are part of a chain should be assigned an ID as described in the [Matching Operator State](#Matching Operator State) section above.
+* **Changing operator chaining:** Operators can be chained together for improved performance. When restoring from a savepoint taken since 1.3.x it is possible to modify chains while preserving state consistency. It is possible a break the chain such that a stateful operator is moved out of the chain. It is also possible to append or inject a new or existing stateful operator into a chain, or to modify the operator order within a chain. However, when upgrading a savepoint to 1.3.x it is paramount that the topology did not change in regards to chaining. All operators that are part of a chain should be assigned an ID as described in the [Matching Operator State](#matching-operator-state) section above.
## Upgrading the Flink Framework Version
@@ -141,7 +141,7 @@ about the steps that we outlined before.
### Preconditions
Before starting the migration, please check that the jobs you are trying to migrate are following the
-best practises for [savepoints]({{ site.baseurl }}/ops/state/savepoints.html). Also, check out the
+best practices for [savepoints]({{ site.baseurl }}/ops/state/savepoints.html). Also, check out the
[API Migration Guides]({{ site.baseurl }}/dev/migration.html) to see if there is any API changes related to migrating
savepoints to newer versions.
@@ -171,9 +171,9 @@ savepoints can be relocated using typical filesystem operations..
First major step in job migration is taking a savepoint of your job running in the older Flink version.
You can do this with the command:
-```sh
+{% highlight shell %}
$ bin/flink savepoint :jobId [:targetDirectory]
-```
+{% endhighlight %}
For more details, please read the [savepoint documentation]({{ site.baseurl }}/ops/state/savepoints.html).
@@ -190,9 +190,9 @@ If you are unfamiliar with installing Flink in your cluster, please read the [de
As the last step of job migration, you resume from the savepoint taken above on the updated cluster. You can do
this with the command:
-```sh
+{% highlight shell %}
$ bin/flink run -s :savepointPath [:runArgs]
-```
+{% endhighlight %}
Again, for more details, please take a look at the [savepoint documentation]({{ site.baseurl }}/ops/state/savepoints.html).
diff --git a/docs/quickstart/java_api_quickstart.md b/docs/quickstart/java_api_quickstart.md
index 9a325911dd2..0d14d840056 100644
--- a/docs/quickstart/java_api_quickstart.md
+++ b/docs/quickstart/java_api_quickstart.md
@@ -55,16 +55,16 @@ Use one of the following commands to __create a project__:
<div class="tab-pane" id="quickstart-script">
{% highlight bash %}
{% if site.is_stable %}
- $ curl https://flink.apache.org/q/quickstart.sh | bash
+ $ curl https://flink.apache.org/q/quickstart.sh | bash -s {{site.version}}
{% else %}
- $ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash
+ $ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s {{site.version}}
{% endif %}
{% endhighlight %}
</div>
{% unless site.is_stable %}
<p style="border-radius: 5px; padding: 5px" class="bg-danger">
- <b>Note</b>: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the commandline. If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For details about this change, please refer to <a href="http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html">Maven official document</a>
+ <b>Note</b>: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For details about this change, please refer to <a href="http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html">Maven official document</a>
</p>
{% endunless %}
</div>
@@ -101,17 +101,19 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer
Some Eclipse bundles include that plugin by default, others require you
to install it manually.
-*A note to Mac OS X users*: The default JVM heapsize for Java mey be too
-small for Flink. You have to manually increase it. In Eclipse, choose
+*A note to Mac OS X users*: The default JVM heapsize for Java may be too
+small for Flink. You have to manually increase it.
+In Eclipse, choose
`Run Configurations -> Arguments` and write into the `VM Arguments`
box: `-Xmx800m`.
+In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
## Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
-that you may have added as dependencoes to the application: `target/<artifact-id>-<version>.jar`.
+that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
@@ -122,15 +124,17 @@ can run time application from the JAR file without additionally specifying the m
Write your application!
If you are writing a streaming application and you are looking for inspiration what to write,
-take a look at the [Stream Processing Application Tutorial]({{ site.baseurl }}/quickstart/run_example_quickstart.html#writing-a-flink-program)
+take a look at the [Stream Processing Application Tutorial]({{ site.baseurl }}/quickstart/run_example_quickstart.html#writing-a-flink-program).
If you are writing a batch processing application and you are looking for inspiration what to write,
-take a look at the [Batch Application Examples]({{ site.baseurl }}/dev/batch/examples.html)
+take a look at the [Batch Application Examples]({{ site.baseurl }}/dev/batch/examples.html).
-For a complete overview over the APIa, have a look at the
+For a complete overview over the APIs, have a look at the
[DataStream API]({{ site.baseurl }}/dev/datastream_api.html) and
[DataSet API]({{ site.baseurl }}/dev/batch/index.html) sections.
+[Here]({{ site.baseurl }}/quickstart/setup_quickstart.html) you can find out how to run an application outside the IDE on a local cluster.
+
If you have any trouble, ask on our
[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-user/).
We are happy to provide help.
diff --git a/docs/quickstart/run_example_quickstart.md b/docs/quickstart/run_example_quickstart.md
index 836bfab151b..1a793382edb 100644
--- a/docs/quickstart/run_example_quickstart.md
+++ b/docs/quickstart/run_example_quickstart.md
@@ -366,7 +366,7 @@ $ bin/flink run -c wikiedits.WikipediaAnalysis path/to/wikiedits-0.1.jar
The output of that command should look similar to this, if everything went according to plan:
-```
+{% highlight plain %}
03/08/2016 15:09:27 Job execution switched to status RUNNING.
03/08/2016 15:09:27 Source: Custom Source(1/1) switched to SCHEDULED
03/08/2016 15:09:27 Source: Custom Source(1/1) switched to DEPLOYING
@@ -374,7 +374,7 @@ The output of that command should look similar to this, if everything went accor
03/08/2016 15:09:27 TriggerWindow(TumblingProcessingTimeWindows(5000), FoldingStateDescriptor{name=window-contents, defaultValue=(,0), serializer=null}, ProcessingTimeTrigger(), WindowedStream.fold(WindowedStream.java:207)) -> Map -> Sink: Unnamed(1/1) switched to DEPLOYING
03/08/2016 15:09:27 TriggerWindow(TumblingProcessingTimeWindows(5000), FoldingStateDescriptor{name=window-contents, defaultValue=(,0), serializer=null}, ProcessingTimeTrigger(), WindowedStream.fold(WindowedStream.java:207)) -> Map -> Sink: Unnamed(1/1) switched to RUNNING
03/08/2016 15:09:27 Source: Custom Source(1/1) switched to RUNNING
-```
+{% endhighlight %}
You can see how the individual operators start running. There are only two, because
the operations after the window get folded into one operation for performance reasons. In Flink
diff --git a/docs/quickstart/scala_api_quickstart.md b/docs/quickstart/scala_api_quickstart.md
index a7b73e322c4..590c461ebad 100644
--- a/docs/quickstart/scala_api_quickstart.md
+++ b/docs/quickstart/scala_api_quickstart.md
@@ -76,9 +76,9 @@ In order to run your project you have to issue the `sbt run` command.
Per default, this will run your job in the same JVM as `sbt` is running.
In order to run your job in a distinct JVM, add the following line to `build.sbt`
-~~~scala
+{% highlight scala %}
fork in run := true
-~~~
+{% endhighlight %}
#### IntelliJ
@@ -98,15 +98,15 @@ In order to import the newly created project into [Eclipse](https://eclipse.org/
These project files can be created via the [sbteclipse](https://github.com/typesafehub/sbteclipse) plugin.
Add the following line to your `PROJECT_DIR/project/plugins.sbt` file:
-~~~bash
+{% highlight bash %}
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
-~~~
+{% endhighlight %}
In `sbt` use the following command to create the Eclipse project files
-~~~bash
+{% highlight bash %}
> eclipse
-~~~
+{% endhighlight %}
Now you can import the project into Eclipse via `File -> Import... -> Existing Projects into Workspace` and then select the project directory.
@@ -140,9 +140,9 @@ Use one of the following commands to __create a project__:
<div class="tab-pane" id="quickstart-script">
{% highlight bash %}
{% if site.is_stable %}
- $ curl https://flink.apache.org/q/quickstart-scala.sh | bash
+ $ curl https://flink.apache.org/q/quickstart-scala.sh | bash -s {{site.version}}
{% else %}
- $ curl https://flink.apache.org/q/quickstart-scala-SNAPSHOT.sh | bash
+ $ curl https://flink.apache.org/q/quickstart-scala-SNAPSHOT.sh | bash -s {{site.version}}
{% endif %}
{% endhighlight %}
</div>
@@ -200,7 +200,7 @@ For Eclipse, you need the following plugins, which you can install from the prov
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
-that you may have added as dependencoes to the application: `target/<artifact-id>-<version>.jar`.
+that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
@@ -221,6 +221,8 @@ For a complete overview over the APIa, have a look at the
[DataStream API]({{ site.baseurl }}/dev/datastream_api.html) and
[DataSet API]({{ site.baseurl }}/dev/batch/index.html) sections.
+[Here]({{ site.baseurl }}/quickstart/setup_quickstart.html) you can find out how to run an application outside the IDE on a local cluster.
+
If you have any trouble, ask on our
[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-user/).
We are happy to provide help.
diff --git a/docs/quickstart/setup_quickstart.md b/docs/quickstart/setup_quickstart.md
index b350d02ab37..f862b4a2add 100644
--- a/docs/quickstart/setup_quickstart.md
+++ b/docs/quickstart/setup_quickstart.md
@@ -30,21 +30,21 @@ Get a Flink example program up and running in a few simple steps.
## Setup: Download and Start Flink
-Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the only requirement is to have a working __Java 7.x__ (or higher) installation. Windows users, please take a look at the [Flink on Windows]({{ site.baseurl }}/start/flink_on_windows.html) guide which describes how to run Flink on Windows for local setups.
+Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the only requirement is to have a working __Java 8.x__ installation. Windows users, please take a look at the [Flink on Windows]({{ site.baseurl }}/start/flink_on_windows.html) guide which describes how to run Flink on Windows for local setups.
You can check the correct installation of Java by issuing the following command:
-~~~bash
+{% highlight bash %}
java -version
-~~~
+{% endhighlight %}
If you have Java 8, the output will look something like this:
-~~~bash
+{% highlight bash %}
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
-~~~
+{% endhighlight %}
{% if site.is_stable %}
<div class="codetabs" markdown="1">
@@ -56,22 +56,22 @@ Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
2. Go to the download directory.
3. Unpack the downloaded archive.
-~~~bash
+{% highlight bash %}
$ cd ~/Downloads # Go to download directory
$ tar xzf flink-*.tgz # Unpack the downloaded archive
$ cd flink-{{site.version}}
-~~~
+{% endhighlight %}
</div>
<div data-lang="MacOS X" markdown="1">
For MacOS X users, Flink can be installed through [Homebrew](https://brew.sh/).
-~~~bash
+{% highlight bash %}
$ brew install apache-flink
...
$ flink --version
Version: 1.2.0, Commit ID: 1c659cf
-~~~
+{% endhighlight %}
</div>
</div>
@@ -80,33 +80,39 @@ Version: 1.2.0, Commit ID: 1c659cf
### Download and Compile
Clone the source code from one of our [repositories](http://flink.apache.org/community.html#source-code), e.g.:
-~~~bash
+{% highlight bash %}
$ git clone https://github.com/apache/flink.git
$ cd flink
$ mvn clean package -DskipTests # this will take up to 10 minutes
$ cd build-target # this is where Flink is installed to
-~~~
+{% endhighlight %}
{% endif %}
### Start a Local Flink Cluster
-~~~bash
+{% highlight bash %}
$ ./bin/start-cluster.sh # Start Flink
-~~~
+{% endhighlight %}
-Check the __JobManager's web frontend__ at [http://localhost:8081](http://localhost:8081) and make sure everything is up and running. The web frontend should report a single available TaskManager instance.
+Check the __Dispatcher's web frontend__ at [http://localhost:8081](http://localhost:8081) and make sure everything is up and running. The web frontend should report a single available TaskManager instance.
-<a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" alt="JobManager: Overview"/></a>
+<a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" alt="Dispatcher: Overview"/></a>
You can also verify that the system is running by checking the log files in the `logs` directory:
-~~~bash
-$ tail log/flink-*-jobmanager-*.log
-INFO ... - Starting JobManager
-INFO ... - Starting JobManager web frontend
-INFO ... - Web frontend listening at 127.0.0.1:8081
-INFO ... - Registered TaskManager at 127.0.0.1 (akka://flink/user/taskmanager)
-~~~
+{% highlight bash %}
+$ tail log/flink-*-standalonesession-*.log
+INFO ... - Rest endpoint listening at localhost:8081
+INFO ... - http://localhost:8081 was granted leadership ...
+INFO ... - Web frontend listening at http://localhost:8081.
+INFO ... - Starting RPC endpoint for StandaloneResourceManager at akka://flink/user/resourcemanager .
+INFO ... - Starting RPC endpoint for StandaloneDispatcher at akka://flink/user/dispatcher .
+INFO ... - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership ...
+INFO ... - Starting the SlotManager.
+INFO ... - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership ...
+INFO ... - Recovering all persisted jobs.
+INFO ... - Registering TaskManager ... under ... at the SlotManager.
+{% endhighlight %}
## Read the Code
@@ -233,38 +239,26 @@ window of processing time, as long as words are floating in.
* First of all, we use **netcat** to start local server via
- ~~~bash
- $ nc -l 9000
- ~~~
+{% highlight bash %}
+$ nc -l 9000
+{% endhighlight %}
* Submit the Flink program:
- ~~~bash
- $ ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
-
- Cluster configuration: Standalone cluster with JobManager at /127.0.0.1:6123
- Using address 127.0.0.1:6123 to connect to JobManager.
- JobManager web interface address http://127.0.0.1:8081
- Starting execution of program
- Submitting job with JobID: 574a10c8debda3dccd0c78a3bde55e1b. Waiting for job completion.
- Connected to JobManager at Actor[akka.tcp://flink@127.0.0.1:6123/user/jobmanager#297388688]
- 11/04/2016 14:04:50 Job execution switched to status RUNNING.
- 11/04/2016 14:04:50 Source: Socket Stream -> Flat Map(1/1) switched to SCHEDULED
- 11/04/2016 14:04:50 Source: Socket Stream -> Flat Map(1/1) switched to DEPLOYING
- 11/04/2016 14:04:50 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to SCHEDULED
- 11/04/2016 14:04:51 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to DEPLOYING
- 11/04/2016 14:04:51 Fast TumblingProcessingTimeWindows(5000) of WindowedStream.main(SocketWindowWordCount.java:79) -> Sink: Unnamed(1/1) switched to RUNNING
- 11/04/2016 14:04:51 Source: Socket Stream -> Flat Map(1/1) switched to RUNNING
- ~~~
+{% highlight bash %}
+$ ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
+Starting execution of program
+
+{% endhighlight %}
The program connects to the socket and waits for input. You can check the web interface to verify that the job is running as expected:
<div class="row">
<div class="col-sm-6">
- <a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-2.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-2.png" alt="JobManager: Overview (cont'd)"/></a>
+ <a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-2.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-2.png" alt="Dispatcher: Overview (cont'd)"/></a>
</div>
<div class="col-sm-6">
- <a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-3.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-3.png" alt="JobManager: Running Jobs"/></a>
+ <a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-3.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-3.png" alt="Dispatcher: Running Jobs"/></a>
</div>
</div>
@@ -273,28 +267,28 @@ window of processing time, as long as words are floating in.
and write some text in `nc` (input is sent to Flink line by line after
hitting <RETURN>):
- ~~~bash
- $ nc -l 9000
- lorem ipsum
- ipsum ipsum ipsum
- bye
- ~~~
+{% highlight bash %}
+$ nc -l 9000
+lorem ipsum
+ipsum ipsum ipsum
+bye
+{% endhighlight %}
The `.out` file will print the counts at the end of each time window as long
as words are floating in, e.g.:
- ~~~bash
- $ tail -f log/flink-*-taskmanager-*.out
- lorem : 1
- bye : 1
- ipsum : 4
- ~~~~
+{% highlight bash %}
+$ tail -f log/flink-*-taskexecutor-*.out
+lorem : 1
+bye : 1
+ipsum : 4
+{% endhighlight %}
To **stop** Flink when you're done type:
- ~~~bash
- $ ./bin/stop-local.sh
- ~~~
+{% highlight bash %}
+$ ./bin/stop-cluster.sh
+{% endhighlight %}
## Next Steps
diff --git a/docs/release-notes/flink-1.5.md b/docs/release-notes/flink-1.5.md
new file mode 100644
index 00000000000..4cee5774cdf
--- /dev/null
+++ b/docs/release-notes/flink-1.5.md
@@ -0,0 +1,84 @@
+---
+title: "Release Notes - Flink 1.5"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.4 and Flink 1.5. Please read these notes carefully if you are planning to upgrade your Flink version to 1.5.
+
+### Update Configuration for Reworked Job Deployment
+
+Flink’s reworked cluster and job deployment component improves the integration with resource managers and enables dynamic resource allocation. One result of these changes is, that you no longer have to specify the number of containers when submitting applications to YARN and Mesos. Flink will automatically determine the number of containers from the parallelism of the application.
+
+Although the deployment logic was completely reworked, we aimed to not unnecessarily change the previous behavior to enable a smooth transition. Nonetheless, there are a few options that you should update in your `conf/flink-conf.yaml` or know about.
+
+* The allocation of TaskManagers with multiple slots is not fully supported yet. Therefore, we recommend to configure TaskManagers with a single slot, i.e., set `taskmanager.numberOfTaskSlots: 1`
+* If you observed any problems with the new deployment mode, you can always switch back to the pre-1.5 behavior by configuring `mode: legacy`.
+
+Please report any problems or possible improvements that you notice to the Flink community, either by posting to a mailing list or by opening a JIRA issue.
+
+*Note*: We plan to remove the legacy mode in the next release.
+
+### Update Configuration for Reworked Network Stack
+
+The changes on the networking stack for credit-based flow control and improved latency affect the configuration of network buffers. In a nutshell, the networking stack can require more memory to run applications. Hence, you might need to adjust the network configuration of your Flink setup.
+
+There are two ways to address problems of job submissions that fail due to lack of network buffers.
+
+* Reduce the number of buffers per channel, i.e., `taskmanager.network.memory.buffers-per-channel` or
+* Increase the amount of TaskManager memory that is used by the network stack, i.e., increase `taskmanager.network.memory.fraction` and/or `taskmanager.network.memory.max`.
+
+Please consult the section about [network buffer configuration]({{ site.baseurl }}/ops/config.html#configuring-the-network-buffers) in the Flink documentation for details. In case you experience issues with the new credit-based flow control mode, you can disable flow control by setting `taskmanager.network.credit-model: false`.
+
+*Note*: We plan to remove the old model and this configuration in the next release.
+
+### Hadoop Classpath Discovery
+
+We removed the automatic Hadoop classpath discovery via the Hadoop binary. If you want Flink to pick up the Hadoop classpath you have to export `HADOOP_CLASSPATH`. On cloud environments and most Hadoop distributions you would do
+
+```
+export HADOOP_CLASSPATH=`hadoop classpath`.
+```
+
+### Breaking Changes of the REST API
+
+In an effort to harmonize, extend, and improve the REST API, a few handlers and return values were changed.
+
+* The jobs overview handler is now registered under `/jobs/overview` (before `/joboverview`) and returns a list of job details instead of the pre-grouped view of running, finished, cancelled and failed jobs.
+* The REST API to cancel a job was changed.
+* The REST API to cancel a job with savepoint was changed.
+
+Please check the [REST API documentation]({{ site.baseurl }}/monitoring/rest_api.html#available-requests) for details.
+
+### Kafka Producer Flushes on Checkpoint by Default
+
+The Flink Kafka Producer now flushes on checkpoints by default. Prior to version 1.5, the behaviour was disabled by default and users had to explicitly call `setFlushOnCheckpoints(true)` on the producer to enable it.
+
+### Updated Kinesis Dependency
+
+The Kinesis dependencies of Flink’s Kinesis connector have been updated to the following versions.
+
+```
+<aws.sdk.version>1.11.319</aws.sdk.version>
+<aws.kinesis-kcl.version>1.9.0</aws.kinesis-kcl.version>
+<aws.kinesis-kpl.version>0.12.9</aws.kinesis-kcl.version>
+```
+
+
+{% top %}
diff --git a/docs/release-notes/flink-1.6.md b/docs/release-notes/flink-1.6.md
new file mode 100644
index 00000000000..31cd3ac9e56
--- /dev/null
+++ b/docs/release-notes/flink-1.6.md
@@ -0,0 +1,26 @@
+---
+title: "Release Notes - Flink 1.6"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+These release notes discuss important aspects, such as configuration, behavior, or dependencies, that changed between Flink 1.5 and Flink 1.6. Please read these notes carefully if you are planning to upgrade your Flink version to 1.6.
+
+
+{% top %}
diff --git a/docs/start/building.md b/docs/start/building.md
index 568c4b83191..654f00bc4db 100644
--- a/docs/start/building.md
+++ b/docs/start/building.md
@@ -38,19 +38,23 @@ To build unit tests use Java 8u51 or above to prevent failures in unit tests tha
To clone from git, enter:
-~~~bash
+{% highlight bash %}
git clone {{ site.github_url }}
-~~~
+{% endhighlight %}
The simplest way of building Flink is by running:
-~~~bash
+{% highlight bash %}
mvn clean install -DskipTests
-~~~
+{% endhighlight %}
This instructs [Maven](http://maven.apache.org) (`mvn`) to first remove all existing builds (`clean`) and then create a new Flink binary (`install`).
-To speed up the build you can skip tests, checkstyle, and JavaDocs: `mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true`.
+To speed up the build you can skip tests, QA plugins, and JavaDocs:
+
+{% highlight bash %}
+mvn clean install -DskipTests -Dfast
+{% endhighlight %}
The default build adds a Flink-specific JAR for Hadoop 2, to allow using Flink with HDFS and YARN.
@@ -66,11 +70,11 @@ It is sufficient to call `mvn clean install -DskipTests` in the root directory o
**Maven 3.3.x**
The build has to be done in two steps: First in the base directory, then in the distribution project:
-~~~bash
+{% highlight bash %}
mvn clean install -DskipTests
cd flink-dist
mvn clean install
-~~~
+{% endhighlight %}
*Note:* To check your Maven version, run `mvn --version`.
@@ -85,17 +89,17 @@ Flink has dependencies to HDFS and YARN which are both dependencies from [Apache
Hadoop is only supported from version 2.4.0 upwards.
You can also specify a specific Hadoop version to build against:
-~~~bash
+{% highlight bash %}
mvn clean install -DskipTests -Dhadoop.version=2.6.1
-~~~
+{% endhighlight %}
### Vendor-specific Versions
To build Flink against a vendor specific Hadoop version, issue the following command:
-~~~bash
+{% highlight bash %}
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.1-cdh5.0.0
-~~~
+{% endhighlight %}
The `-Pvendor-repos` activates a Maven [build profile](http://maven.apache.org/guides/introduction/introduction-to-profiles.html) that includes the repositories of popular Hadoop vendors such as Cloudera, Hortonworks, or MapR.
@@ -119,12 +123,12 @@ If your home directory is encrypted you might encounter a `java.io.IOException:
The workaround is to add:
-~~~xml
+{% highlight xml %}
<args>
<arg>-Xmax-classfile-name</arg>
<arg>128</arg>
</args>
-~~~
+{% endhighlight %}
in the compiler configuration of the `pom.xml` file of the module causing the error. For example, if the error appears in the `flink-yarn` module, the above code should be added under the `<configuration>` tag of `scala-maven-plugin`. See [this issue](https://issues.apache.org/jira/browse/FLINK-2003) for more information.
diff --git a/docs/start/dependencies.md b/docs/start/dependencies.md
index 1375c6f30f8..564636863f5 100644
--- a/docs/start/dependencies.md
+++ b/docs/start/dependencies.md
@@ -32,7 +32,7 @@ runtime library must be available as well.
As with most systems that run user-defined applications, there are two broad categories of dependencies and libraries in Flink:
- - **Flink Core Dependenies**: Flink itself consists of a set of classes and dependencies that are needed to run the system, for example
+ - **Flink Core Dependencies**: Flink itself consists of a set of classes and dependencies that are needed to run the system, for example
coordination, networking, checkpoints, failover, APIs, operations (such as windowing), resource management, etc.
The set of all these classes and dependencies forms the core of Flink's runtime and must be present when a Flink
application is started.
@@ -189,7 +189,7 @@ If you need Hadoop dependencies during testing or development inside the IDE (fo
these dependencies similar to the scope of the dependencies to *test* or to *provided*.
-## Appendix: Template for bulding a Jar with Dependencies
+## Appendix: Template for building a Jar with Dependencies
To build an application JAR that contains all dependencies required for declared connectors and libraries,
you can use the following shade plugin definition:
@@ -229,7 +229,7 @@ you can use the following shade plugin definition:
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <mainClass>my.prorgams.main.clazz</mainClass>
+ <mainClass>my.programs.main.clazz</mainClass>
</transformer>
</transformers>
</configuration>
diff --git a/docs/start/flink_on_windows.md b/docs/start/flink_on_windows.md
index e2f422817fd..b8dcbf7db29 100644
--- a/docs/start/flink_on_windows.md
+++ b/docs/start/flink_on_windows.md
@@ -30,14 +30,14 @@ To start Flink in from the *Windows Command Line*, open the command window, navi
Note: The ``bin`` folder of your Java Runtime Environment must be included in Window's ``%PATH%`` variable. Follow this [guide](http://www.java.com/en/download/help/path.xml) to add Java to the ``%PATH%`` variable.
-~~~bash
+{% highlight bash %}
$ cd flink
$ cd bin
$ start-cluster.bat
Starting a local cluster with one JobManager process and one TaskManager process.
You can terminate the processes via CTRL-C in the spawned shell windows.
Web interface by default on http://localhost:8081/.
-~~~
+{% endhighlight %}
After that, you need to open a second terminal to run jobs using `flink.bat`.
@@ -47,11 +47,11 @@ After that, you need to open a second terminal to run jobs using `flink.bat`.
With *Cygwin* you need to start the Cygwin Terminal, navigate to your Flink directory and run the `start-cluster.sh` script:
-~~~bash
+{% highlight bash %}
$ cd flink
$ bin/start-cluster.sh
Starting cluster.
-~~~
+{% endhighlight %}
{% top %}
@@ -59,9 +59,9 @@ Starting cluster.
If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
-~~~bash
+{% highlight bash %}
c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
-~~~
+{% endhighlight %}
This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
@@ -69,18 +69,18 @@ This error occurs because git is automatically transforming UNIX line endings to
2. Determine your home directory by entering
- ~~~bash
+ {% highlight bash %}
cd; pwd
- ~~~
+ {% endhighlight %}
This will return a path under the Cygwin root path.
3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
-~~~bash
+{% highlight bash %}
export SHELLOPTS
set -o igncr
-~~~
+{% endhighlight %}
Save the file and open a new bash shell.
diff --git a/flink-annotations/pom.xml b/flink-annotations/pom.xml
index f28b32c5ee4..adb286467f1 100644
--- a/flink-annotations/pom.xml
+++ b/flink-annotations/pom.xml
@@ -25,7 +25,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parent</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-annotations/src/main/java/org/apache/flink/annotation/docs/Documentation.java b/flink-annotations/src/main/java/org/apache/flink/annotation/docs/Documentation.java
new file mode 100644
index 00000000000..c193c4e0554
--- /dev/null
+++ b/flink-annotations/src/main/java/org/apache/flink/annotation/docs/Documentation.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.annotation.docs;
+
+import org.apache.flink.annotation.Internal;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Collection of annotations to modify the behavior of the documentation generators.
+ */
+public final class Documentation {
+
+ /**
+ * Annotation used on config option fields to override the documented default.
+ */
+ @Target(ElementType.FIELD)
+ @Retention(RetentionPolicy.RUNTIME)
+ @Internal
+ public @interface OverrideDefault {
+ String value();
+ }
+
+ /**
+ * Annotation used on config option fields to include them in the "Common Options" section.
+ *
+ * <p>The {@link CommonOption#position()} argument controls the position in the generated table, with lower values
+ * being placed at the top. Fields with the same position are sorted alphabetically by key.
+ */
+ @Target(ElementType.FIELD)
+ @Retention(RetentionPolicy.RUNTIME)
+ @Internal
+ public @interface CommonOption {
+ int POSITION_MEMORY = 10;
+ int POSITION_PARALLELISM_SLOTS = 20;
+ int POSITION_FAULT_TOLERANCE = 30;
+ int POSITION_HIGH_AVAILABILITY = 40;
+ int POSITION_SECURITY = 50;
+
+ int position() default Integer.MAX_VALUE;
+ }
+
+ private Documentation(){
+ }
+}
diff --git a/flink-clients/pom.xml b/flink-clients/pom.xml
index c96f2754bd9..dafae8189c3 100644
--- a/flink-clients/pom.xml
+++ b/flink-clients/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parent</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
@@ -95,18 +95,6 @@ under the License.
-->
<build>
<plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jar-plugin</artifactId>
- <executions>
- <execution>
- <goals>
- <goal>test-jar</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
-
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version><!--$NO-MVN-MAN-VER$-->
diff --git a/flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java b/flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
index 01c281fdc6b..4e4993a88db 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java
@@ -28,6 +28,7 @@
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.CoreOptions;
import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
import org.apache.flink.optimizer.DataStatistics;
import org.apache.flink.optimizer.Optimizer;
import org.apache.flink.optimizer.dag.DataSinkNode;
@@ -39,6 +40,7 @@
import org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster;
import org.apache.flink.runtime.minicluster.MiniCluster;
import org.apache.flink.runtime.minicluster.MiniClusterConfiguration;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
import java.util.List;
@@ -136,10 +138,10 @@ private JobExecutorService createJobExecutorService(Configuration configuration)
configuration.getInteger(
ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
ConfigConstants.DEFAULT_LOCAL_NUMBER_TASK_MANAGER))
- .setRpcServiceSharing(MiniClusterConfiguration.RpcServiceSharing.SHARED)
+ .setRpcServiceSharing(RpcServiceSharing.SHARED)
.setNumSlotsPerTaskManager(
configuration.getInteger(
- ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 1))
+ TaskManagerOptions.NUM_TASK_SLOTS, 1))
.build();
final MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration);
@@ -220,7 +222,7 @@ public JobExecutionResult executePlan(Plan plan) throws Exception {
try {
// TODO: Set job's default parallelism to max number of slots
- final int slotsPerTaskManager = jobExecutorServiceConfiguration.getInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, taskManagerNumSlots);
+ final int slotsPerTaskManager = jobExecutorServiceConfiguration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, taskManagerNumSlots);
final int numTaskManagers = jobExecutorServiceConfiguration.getInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
plan.setDefaultParallelism(slotsPerTaskManager * numTaskManagers);
@@ -265,7 +267,7 @@ public void endSession(JobID jobID) throws Exception {
private Configuration createConfiguration() {
Configuration newConfiguration = new Configuration();
- newConfiguration.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, getTaskManagerNumSlots());
+ newConfiguration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, getTaskManagerNumSlots());
newConfiguration.setBoolean(CoreOptions.FILESYTEM_DEFAULT_OVERRIDE, isDefaultOverwriteFiles());
newConfiguration.addAll(baseConfiguration);
diff --git a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
index 7745ca0ac34..e2a260c5478 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
@@ -81,6 +81,7 @@
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
import scala.concurrent.duration.FiniteDuration;
@@ -165,6 +166,10 @@ public Configuration getConfiguration() {
return copiedConfiguration;
}
+ public Options getCustomCommandLineOptions() {
+ return customCommandLineOptions;
+ }
+
// --------------------------------------------------------------------------------------------
// Execute Actions
// --------------------------------------------------------------------------------------------
@@ -277,8 +282,8 @@ protected void run(String[] args) throws Exception {
if (clusterId == null && !client.isDetached()) {
// terminate the cluster only if we have started it before and if it's not detached
try {
- clusterDescriptor.terminateCluster(client.getClusterId());
- } catch (FlinkException e) {
+ client.shutDownCluster();
+ } catch (final Exception e) {
LOG.info("Could not properly terminate the Flink cluster.", e);
}
}
@@ -392,16 +397,19 @@ protected void list(String[] args) throws Exception {
return;
}
- final boolean running;
- final boolean scheduled;
+ final boolean showRunning;
+ final boolean showScheduled;
+ final boolean showAll;
// print running and scheduled jobs if not option supplied
- if (!listOptions.getRunning() && !listOptions.getScheduled()) {
- running = true;
- scheduled = true;
+ if (!listOptions.showRunning() && !listOptions.showScheduled() && !listOptions.showAll()) {
+ showRunning = true;
+ showScheduled = true;
+ showAll = false;
} else {
- running = listOptions.getRunning();
- scheduled = listOptions.getScheduled();
+ showRunning = listOptions.showRunning();
+ showScheduled = listOptions.showScheduled();
+ showAll = listOptions.showAll();
}
final CustomCommandLine<?> activeCommandLine = getActiveCustomCommandLine(commandLine);
@@ -409,14 +417,15 @@ protected void list(String[] args) throws Exception {
runClusterAction(
activeCommandLine,
commandLine,
- clusterClient -> listJobs(clusterClient, running, scheduled));
+ clusterClient -> listJobs(clusterClient, showRunning, showScheduled, showAll));
}
private <T> void listJobs(
ClusterClient<T> clusterClient,
- boolean running,
- boolean scheduled) throws FlinkException {
+ boolean showRunning,
+ boolean showScheduled,
+ boolean showAll) throws FlinkException {
Collection<JobStatusMessage> jobDetails;
try {
CompletableFuture<Collection<JobStatusMessage>> jobDetailsFuture = clusterClient.listJobs();
@@ -431,49 +440,63 @@ protected void list(String[] args) throws Exception {
LOG.info("Successfully retrieved list of jobs");
- SimpleDateFormat dateFormat = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss");
- Comparator<JobStatusMessage> startTimeComparator = (o1, o2) -> (int) (o1.getStartTime() - o2.getStartTime());
-
final List<JobStatusMessage> runningJobs = new ArrayList<>();
final List<JobStatusMessage> scheduledJobs = new ArrayList<>();
+ final List<JobStatusMessage> terminatedJobs = new ArrayList<>();
jobDetails.forEach(details -> {
if (details.getJobState() == JobStatus.CREATED) {
scheduledJobs.add(details);
- } else {
+ } else if (!details.getJobState().isGloballyTerminalState()) {
runningJobs.add(details);
+ } else {
+ terminatedJobs.add(details);
}
});
- if (running) {
+ if (showRunning || showAll) {
if (runningJobs.size() == 0) {
System.out.println("No running jobs.");
}
else {
- runningJobs.sort(startTimeComparator);
-
System.out.println("------------------ Running/Restarting Jobs -------------------");
- for (JobStatusMessage runningJob : runningJobs) {
- System.out.println(dateFormat.format(new Date(runningJob.getStartTime()))
- + " : " + runningJob.getJobId() + " : " + runningJob.getJobName() + " (" + runningJob.getJobState() + ")");
- }
+ printJobStatusMessages(runningJobs);
System.out.println("--------------------------------------------------------------");
}
}
- if (scheduled) {
+ if (showScheduled || showAll) {
if (scheduledJobs.size() == 0) {
System.out.println("No scheduled jobs.");
}
else {
- scheduledJobs.sort(startTimeComparator);
-
System.out.println("----------------------- Scheduled Jobs -----------------------");
- for (JobStatusMessage scheduledJob : scheduledJobs) {
- System.out.println(dateFormat.format(new Date(scheduledJob.getStartTime()))
- + " : " + scheduledJob.getJobId() + " : " + scheduledJob.getJobName());
- }
+ printJobStatusMessages(scheduledJobs);
System.out.println("--------------------------------------------------------------");
}
}
+ if (showAll) {
+ if (terminatedJobs.size() != 0) {
+ System.out.println("---------------------- Terminated Jobs -----------------------");
+ printJobStatusMessages(terminatedJobs);
+ System.out.println("--------------------------------------------------------------");
+ }
+ }
+ }
+
+ private static void printJobStatusMessages(List<JobStatusMessage> jobs) {
+ SimpleDateFormat dateFormat = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss");
+ Comparator<JobStatusMessage> startTimeComparator = (o1, o2) -> (int) (o1.getStartTime() - o2.getStartTime());
+ Comparator<Map.Entry<JobStatus, List<JobStatusMessage>>> statusComparator =
+ (o1, o2) -> String.CASE_INSENSITIVE_ORDER.compare(o1.getKey().toString(), o2.getKey().toString());
+
+ Map<JobStatus, List<JobStatusMessage>> jobsByState = jobs.stream().collect(Collectors.groupingBy(JobStatusMessage::getJobState));
+ jobsByState.entrySet().stream()
+ .sorted(statusComparator)
+ .map(Map.Entry::getValue).flatMap(List::stream).sorted(startTimeComparator)
+ .forEachOrdered(job -> {
+ System.out.println(dateFormat.format(new Date(job.getStartTime()))
+ + " : " + job.getJobId() + " : " + job.getJobName()
+ + " (" + job.getJobState() + ")");
+ });
}
/**
@@ -850,7 +873,7 @@ else if (!jarFile.isFile()) {
* @return The return code for the process.
*/
private static int handleArgException(CliArgsException e) {
- LOG.error("Invalid command line arguments. " + (e.getMessage() == null ? "" : e.getMessage()));
+ LOG.error("Invalid command line arguments.", e);
System.out.println(e.getMessage());
System.out.println();
@@ -865,6 +888,7 @@ private static int handleArgException(CliArgsException e) {
* @return The return code for the process.
*/
private static int handleParametrizationException(ProgramParametrizationException e) {
+ LOG.error("Program has not been parametrized properly.", e);
System.err.println(e.getMessage());
return 1;
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java
index 1588aac1bbe..0adb8cf75b2 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java
@@ -94,6 +94,9 @@
static final Option SCHEDULED_OPTION = new Option("s", "scheduled", false,
"Show only scheduled programs and their JobIDs");
+ static final Option ALL_OPTION = new Option("a", "all", false,
+ "Show all programs and their JobIDs");
+
static final Option ZOOKEEPER_NAMESPACE_OPTION = new Option("z", "zookeeperNamespace", true,
"Namespace to create the Zookeeper sub-paths for high availability mode");
@@ -193,6 +196,7 @@ static Options getInfoCommandOptions() {
static Options getListCommandOptions() {
Options options = buildGeneralOptions(new Options());
+ options.addOption(ALL_OPTION);
options.addOption(RUNNING_OPTION);
return options.addOption(SCHEDULED_OPTION);
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/cli/ListOptions.java b/flink-clients/src/main/java/org/apache/flink/client/cli/ListOptions.java
index 7ae00cfce41..f4c124639ad 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/cli/ListOptions.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/cli/ListOptions.java
@@ -20,6 +20,7 @@
import org.apache.commons.cli.CommandLine;
+import static org.apache.flink.client.cli.CliFrontendParser.ALL_OPTION;
import static org.apache.flink.client.cli.CliFrontendParser.RUNNING_OPTION;
import static org.apache.flink.client.cli.CliFrontendParser.SCHEDULED_OPTION;
@@ -28,20 +29,26 @@
*/
public class ListOptions extends CommandLineOptions {
- private final boolean running;
- private final boolean scheduled;
+ private final boolean showRunning;
+ private final boolean showScheduled;
+ private final boolean showAll;
public ListOptions(CommandLine line) {
super(line);
- this.running = line.hasOption(RUNNING_OPTION.getOpt());
- this.scheduled = line.hasOption(SCHEDULED_OPTION.getOpt());
+ this.showAll = line.hasOption(ALL_OPTION.getOpt());
+ this.showRunning = line.hasOption(RUNNING_OPTION.getOpt());
+ this.showScheduled = line.hasOption(SCHEDULED_OPTION.getOpt());
}
- public boolean getRunning() {
- return running;
+ public boolean showRunning() {
+ return showRunning;
}
- public boolean getScheduled() {
- return scheduled;
+ public boolean showScheduled() {
+ return showScheduled;
+ }
+
+ public boolean showAll() {
+ return showAll;
}
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterDescriptor.java b/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterDescriptor.java
index f9f5d4b5e10..e6b3922c91d 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterDescriptor.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterDescriptor.java
@@ -72,5 +72,5 @@
* @param clusterId identifying the cluster to shut down
* @throws FlinkException if the cluster could not be terminated
*/
- void terminateCluster(T clusterId) throws FlinkException;
+ void killCluster(T clusterId) throws FlinkException;
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterSpecification.java b/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterSpecification.java
index 8650cab7806..72975d8b9b2 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterSpecification.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/deployment/ClusterSpecification.java
@@ -18,9 +18,8 @@
package org.apache.flink.client.deployment;
-import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.ConfigurationUtils;
import org.apache.flink.configuration.TaskManagerOptions;
/**
@@ -66,10 +65,10 @@ public String toString() {
}
public static ClusterSpecification fromConfiguration(Configuration configuration) {
- int slots = configuration.getInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 1);
+ int slots = configuration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, 1);
- int jobManagerMemoryMb = configuration.getInteger(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY);
- int taskManagerMemoryMb = configuration.getInteger(TaskManagerOptions.TASK_MANAGER_HEAP_MEMORY);
+ int jobManagerMemoryMb = ConfigurationUtils.getJobManagerHeapMemory(configuration).getMebiBytes();
+ int taskManagerMemoryMb = ConfigurationUtils.getTaskManagerHeapMemory(configuration).getMebiBytes();
return new ClusterSpecificationBuilder()
.setMasterMemoryMB(jobManagerMemoryMb)
diff --git a/flink-clients/src/main/java/org/apache/flink/client/deployment/LegacyStandaloneClusterDescriptor.java b/flink-clients/src/main/java/org/apache/flink/client/deployment/LegacyStandaloneClusterDescriptor.java
index 21e020ced55..b44897054d6 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/deployment/LegacyStandaloneClusterDescriptor.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/deployment/LegacyStandaloneClusterDescriptor.java
@@ -65,7 +65,7 @@ public StandaloneClusterClient deployJobCluster(
}
@Override
- public void terminateCluster(StandaloneClusterId clusterId) throws FlinkException {
+ public void killCluster(StandaloneClusterId clusterId) throws FlinkException {
throw new UnsupportedOperationException("Cannot terminate standalone clusters.");
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/deployment/StandaloneClusterDescriptor.java b/flink-clients/src/main/java/org/apache/flink/client/deployment/StandaloneClusterDescriptor.java
index c4bcde690a6..bdf8faf9cc5 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/deployment/StandaloneClusterDescriptor.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/deployment/StandaloneClusterDescriptor.java
@@ -66,7 +66,7 @@ public String getClusterDescription() {
}
@Override
- public void terminateCluster(StandaloneClusterId clusterId) throws FlinkException {
+ public void killCluster(StandaloneClusterId clusterId) throws FlinkException {
throw new UnsupportedOperationException("Cannot terminate a standalone cluster.");
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/ActorSystemLoader.java b/flink-clients/src/main/java/org/apache/flink/client/program/ActorSystemLoader.java
new file mode 100644
index 00000000000..eadfc426465
--- /dev/null
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/ActorSystemLoader.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.util.FlinkException;
+
+import akka.actor.ActorSystem;
+
+/**
+ * Interface which allows to load an {@link ActorSystem}.
+ */
+public interface ActorSystemLoader extends AutoCloseable {
+
+ /**
+ * Get an {@link ActorSystem}.
+ *
+ * @return {@link ActorSystem}
+ * @throws FlinkException
+ */
+ ActorSystem get() throws FlinkException;
+}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java b/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java
index 2ef0b2ec65a..373267ffc34 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java
@@ -104,8 +104,8 @@
/** The optimizer used in the optimization of batch programs. */
final Optimizer compiler;
- /** The actor system used to communicate with the JobManager. Lazily initialized upon first use */
- protected final LazyActorSystemLoader actorSystemLoader;
+ /** The actor system used to communicate with the JobManager. */
+ protected final ActorSystemLoader actorSystemLoader;
/** Configuration of the client. */
protected final Configuration flinkConfig;
@@ -171,7 +171,10 @@ public ClusterClient(Configuration flinkConfig) throws Exception {
* @param highAvailabilityServices HighAvailabilityServices to use for leader retrieval
* @param sharedHaServices true if the HighAvailabilityServices are shared and must not be shut down
*/
- public ClusterClient(Configuration flinkConfig, HighAvailabilityServices highAvailabilityServices, boolean sharedHaServices) {
+ public ClusterClient(
+ Configuration flinkConfig,
+ HighAvailabilityServices highAvailabilityServices,
+ boolean sharedHaServices) {
this.flinkConfig = Preconditions.checkNotNull(flinkConfig);
this.compiler = new Optimizer(new DataStatistics(), new DefaultCostEstimator(), flinkConfig);
@@ -188,6 +191,23 @@ public ClusterClient(Configuration flinkConfig, HighAvailabilityServices highAva
this.sharedHaServices = sharedHaServices;
}
+ public ClusterClient(
+ Configuration flinkConfig,
+ HighAvailabilityServices highAvailabilityServices,
+ boolean sharedHaServices,
+ ActorSystemLoader actorSystemLoader) {
+ this.flinkConfig = Preconditions.checkNotNull(flinkConfig);
+ this.compiler = new Optimizer(new DataStatistics(), new DefaultCostEstimator(), flinkConfig);
+
+ this.timeout = AkkaUtils.getClientTimeout(flinkConfig);
+ this.lookupTimeout = AkkaUtils.getLookupTimeout(flinkConfig);
+
+ this.actorSystemLoader = Preconditions.checkNotNull(actorSystemLoader);
+
+ this.highAvailabilityServices = Preconditions.checkNotNull(highAvailabilityServices);
+ this.sharedHaServices = sharedHaServices;
+ }
+
// ------------------------------------------------------------------------
// Startup & Shutdown
// ------------------------------------------------------------------------
@@ -195,7 +215,7 @@ public ClusterClient(Configuration flinkConfig, HighAvailabilityServices highAva
/**
* Utility class to lazily instantiate an {@link ActorSystem}.
*/
- protected static class LazyActorSystemLoader {
+ protected static class LazyActorSystemLoader implements ActorSystemLoader {
private final Logger log;
@@ -226,7 +246,8 @@ public boolean isLoaded() {
return actorSystem != null;
}
- public void shutdown() {
+ @Override
+ public void close() throws Exception {
if (isLoaded()) {
actorSystem.shutdown();
actorSystem.awaitTermination();
@@ -239,6 +260,7 @@ public void shutdown() {
* @return ActorSystem
* @throws Exception if the ActorSystem could not be created
*/
+ @Override
public ActorSystem get() throws FlinkException {
if (!isLoaded()) {
@@ -276,7 +298,7 @@ public ActorSystem get() throws FlinkException {
*/
public void shutdown() throws Exception {
synchronized (this) {
- actorSystemLoader.shutdown();
+ actorSystemLoader.close();
if (!sharedHaServices && highAvailabilityServices != null) {
highAvailabilityServices.close();
@@ -481,7 +503,7 @@ public JobExecutionResult run(JobGraph jobGraph, ClassLoader classLoader) throws
actorSystem = actorSystemLoader.get();
} catch (FlinkException fe) {
throw new ProgramInvocationException("Could not start the ActorSystem needed to talk to the " +
- "JobManager.", fe);
+ "JobManager.", jobGraph.getJobID(), fe);
}
try {
@@ -497,7 +519,7 @@ public JobExecutionResult run(JobGraph jobGraph, ClassLoader classLoader) throws
return lastJobExecutionResult;
} catch (JobExecutionException e) {
- throw new ProgramInvocationException("The program execution failed: " + e.getMessage(), e);
+ throw new ProgramInvocationException("The program execution failed: " + e.getMessage(), jobGraph.getJobID(), e);
}
}
@@ -516,7 +538,8 @@ public JobSubmissionResult runDetached(JobGraph jobGraph, ClassLoader classLoade
try {
jobManagerGateway = getJobManagerGateway();
} catch (Exception e) {
- throw new ProgramInvocationException("Failed to retrieve the JobManager gateway.", e);
+ throw new ProgramInvocationException("Failed to retrieve the JobManager gateway.",
+ jobGraph.getJobID(), e);
}
try {
@@ -529,7 +552,8 @@ public JobSubmissionResult runDetached(JobGraph jobGraph, ClassLoader classLoade
classLoader);
return new JobSubmissionResult(jobGraph.getJobID());
} catch (JobExecutionException e) {
- throw new ProgramInvocationException("The program execution failed: " + e.getMessage(), e);
+ throw new ProgramInvocationException("The program execution failed: " + e.getMessage(),
+ jobGraph.getJobID(), e);
}
}
@@ -1036,4 +1060,7 @@ public abstract JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader cla
throw new UnsupportedOperationException("The " + getClass().getSimpleName() + " does not support rescaling.");
}
+ public void shutDownCluster() {
+ throw new UnsupportedOperationException("The " + getClass().getSimpleName() + " does not support shutDownCluster.");
+ }
}
diff --git a/flink-tests/src/test/java/org/apache/flink/test/checkpointing/RocksDbBackendEventTimeWindowCheckpointingITCase.java b/flink-clients/src/main/java/org/apache/flink/client/program/DefaultActorSystemLoader.java
similarity index 61%
rename from flink-tests/src/test/java/org/apache/flink/test/checkpointing/RocksDbBackendEventTimeWindowCheckpointingITCase.java
rename to flink-clients/src/main/java/org/apache/flink/client/program/DefaultActorSystemLoader.java
index e6d5b9e103c..1f80e9e2874 100644
--- a/flink-tests/src/test/java/org/apache/flink/test/checkpointing/RocksDbBackendEventTimeWindowCheckpointingITCase.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/DefaultActorSystemLoader.java
@@ -16,35 +16,32 @@
* limitations under the License.
*/
-package org.apache.flink.test.checkpointing;
+package org.apache.flink.client.program;
+
+import akka.actor.ActorSystem;
+
+import javax.annotation.Nonnull;
/**
- * Integration tests for fully synchronous RocksDB backend.
+ * {@link ActorSystemLoader} which simply returns the given {@link ActorSystem} at
+ * construction time.
*/
-public class RocksDbBackendEventTimeWindowCheckpointingITCase extends AbstractEventTimeWindowCheckpointingITCase {
-
- @Override
- protected StateBackendEnum getStateBackend() {
- return StateBackendEnum.ROCKSDB_FULLY_ASYNC;
- }
+public class DefaultActorSystemLoader implements ActorSystemLoader {
- @Override
- protected int numElementsPerKey() {
- return 3000;
- }
+ @Nonnull
+ private final ActorSystem actorSystem;
- @Override
- protected int windowSize() {
- return 1000;
+ public DefaultActorSystemLoader(@Nonnull ActorSystem actorSystem) {
+ this.actorSystem = actorSystem;
}
@Override
- protected int windowSlide() {
- return 100;
+ public ActorSystem get() {
+ return actorSystem;
}
@Override
- protected int numKeys() {
- return 100;
+ public void close() throws Exception {
+ // noop
}
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/MiniClusterClient.java b/flink-clients/src/main/java/org/apache/flink/client/program/MiniClusterClient.java
index faf96ec2965..81cf784441d 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/MiniClusterClient.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/MiniClusterClient.java
@@ -77,9 +77,7 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
} catch (InterruptedException | ExecutionException e) {
ExceptionUtils.checkInterrupted(e);
- throw new ProgramInvocationException(
- String.format("Could not run job %s in detached mode.", jobGraph.getJobID()),
- e);
+ throw new ProgramInvocationException("Could not run job in detached mode.", jobGraph.getJobID(), e);
}
} else {
final CompletableFuture<JobResult> jobResultFuture = jobSubmissionResultFuture.thenCompose(
@@ -91,17 +89,15 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
} catch (InterruptedException | ExecutionException e) {
ExceptionUtils.checkInterrupted(e);
- throw new ProgramInvocationException(
- String.format("Could not run job %s.", jobGraph.getJobID()),
- e);
+ throw new ProgramInvocationException("Could not run job", jobGraph.getJobID(), e);
}
try {
return jobResult.toJobExecutionResult(classLoader);
} catch (JobResult.WrappedJobException e) {
- throw new ProgramInvocationException(e.getCause());
+ throw new ProgramInvocationException("Job failed", jobGraph.getJobID(), e.getCause());
} catch (IOException | ClassNotFoundException e) {
- throw new ProgramInvocationException(e);
+ throw new ProgramInvocationException("Job failed", jobGraph.getJobID(), e);
}
}
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
index d81cacbb4f4..bb3d294330e 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
@@ -223,7 +223,7 @@ public PackagedProgram(File jarFile, List<URL> classpaths, @Nullable String entr
}
}
- PackagedProgram(Class<?> entryPointClass, String... args) throws ProgramInvocationException {
+ public PackagedProgram(Class<?> entryPointClass, String... args) throws ProgramInvocationException {
this.jarFile = null;
this.args = args == null ? new String[0] : args;
@@ -294,7 +294,7 @@ public JobWithJars getPlanWithoutJars() throws ProgramInvocationException {
return new JobWithJars(getPlan(), Collections.<URL>emptyList(), classpaths, userCodeClassLoader);
} else {
throw new ProgramInvocationException("Cannot create a " + JobWithJars.class.getSimpleName() +
- " for a program that is using the interactive mode.");
+ " for a program that is using the interactive mode.", getPlan().getJobId());
}
}
@@ -309,7 +309,7 @@ public JobWithJars getPlanWithJars() throws ProgramInvocationException {
return new JobWithJars(getPlan(), getAllLibraries(), classpaths, userCodeClassLoader);
} else {
throw new ProgramInvocationException("Cannot create a " + JobWithJars.class.getSimpleName() +
- " for a program that is using the interactive mode.");
+ " for a program that is using the interactive mode.", getPlan().getJobId());
}
}
@@ -341,12 +341,12 @@ else if (isUsingInteractiveMode()) {
}
catch (Throwable t) {
// the invocation gets aborted with the preview plan
- if (env.previewPlan != null) {
- previewPlan = env.previewPlan;
- } else if (env.preview != null) {
- return env.preview;
- } else {
- throw new ProgramInvocationException("The program caused an error: ", t);
+ if (env.previewPlan == null) {
+ if (env.preview != null) {
+ return env.preview;
+ } else {
+ throw new ProgramInvocationException("The program caused an error: ", getPlan().getJobId(), t);
+ }
}
}
finally {
@@ -357,7 +357,8 @@ else if (isUsingInteractiveMode()) {
previewPlan = env.previewPlan;
} else {
throw new ProgramInvocationException(
- "The program plan could not be fetched. The program silently swallowed the control flow exceptions.");
+ "The program plan could not be fetched. The program silently swallowed the control flow exceptions.",
+ getPlan().getJobId());
}
}
else {
@@ -695,7 +696,9 @@ private static Plan createPlanFromProgram(Program program, String[] options) thr
for (int i = 0; i < containedJarFileEntries.size(); i++) {
final JarEntry entry = containedJarFileEntries.get(i);
String name = entry.getName();
- name = name.replace(File.separatorChar, '_');
+ // '/' as in case of zip, jar
+ // java.util.zip.ZipEntry#isDirectory always looks only for '/' not for File.separator
+ name = name.replace('/', '_');
File tempFile;
try {
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramUtils.java b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramUtils.java
index 5765d1ffa5a..94fc109c47b 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramUtils.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramUtils.java
@@ -90,7 +90,7 @@ public static JobGraph createJobGraph(
try {
jobGraph.addJar(new Path(url.toURI()));
} catch (URISyntaxException e) {
- throw new ProgramInvocationException("Invalid URL for jar file: " + url + '.', e);
+ throw new ProgramInvocationException("Invalid URL for jar file: " + url + '.', jobGraph.getJobID(), e);
}
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/ProgramInvocationException.java b/flink-clients/src/main/java/org/apache/flink/client/program/ProgramInvocationException.java
index ee58227fadf..ac4e3ce2b69 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/ProgramInvocationException.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/ProgramInvocationException.java
@@ -18,6 +18,8 @@
package org.apache.flink.client.program;
+import org.apache.flink.api.common.JobID;
+
/**
* Exception used to indicate that there is an error during the invocation of a Flink program.
*/
@@ -37,6 +39,18 @@ public ProgramInvocationException(String message) {
super(message);
}
+ /**
+ * Creates a <tt>ProgramInvocationException</tt> with the given message which contains job id.
+ *
+ * @param message
+ * The additional message.
+ * @param jobID
+ * ID of failed job.
+ */
+ public ProgramInvocationException(String message, JobID jobID) {
+ super(message + " (JobID: " + jobID + ")");
+ }
+
/**
* Creates a <tt>ProgramInvocationException</tt> for the given exception.
*
@@ -59,4 +73,19 @@ public ProgramInvocationException(Throwable cause) {
public ProgramInvocationException(String message, Throwable cause) {
super(message, cause);
}
+
+ /**
+ * Creates a <tt>ProgramInvocationException</tt> for the given exception with an
+ * additional message which contains job id.
+ *
+ * @param message
+ * The additional message.
+ * @param jobID
+ * ID of failed job.
+ * @param cause
+ * The exception that causes the program invocation to fail.
+ */
+ public ProgramInvocationException(String message, JobID jobID, Throwable cause) {
+ super(message + " (JobID: " + jobID + ")", cause);
+ }
}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/StandaloneClusterClient.java b/flink-clients/src/main/java/org/apache/flink/client/program/StandaloneClusterClient.java
index e502add468c..caee34f4b5e 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/StandaloneClusterClient.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/StandaloneClusterClient.java
@@ -51,6 +51,10 @@ public StandaloneClusterClient(Configuration config, HighAvailabilityServices hi
super(config, highAvailabilityServices, sharedHaServices);
}
+ public StandaloneClusterClient(Configuration config, HighAvailabilityServices highAvailabilityServices, boolean sharedHaServices, ActorSystemLoader actorSystemLoader) {
+ super(config, highAvailabilityServices, sharedHaServices, actorSystemLoader);
+ }
+
@Override
public void waitForClusterToBeReady() {}
diff --git a/flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java b/flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java
index 3d50e93d3a1..935a07faf89 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java
@@ -22,15 +22,16 @@
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobSubmissionResult;
import org.apache.flink.api.common.accumulators.AccumulatorHelper;
+import org.apache.flink.api.common.cache.DistributedCache;
import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.client.program.NewClusterClient;
import org.apache.flink.client.program.ProgramInvocationException;
import org.apache.flink.client.program.rest.retry.ExponentialWaitStrategy;
import org.apache.flink.client.program.rest.retry.WaitStrategy;
import org.apache.flink.configuration.Configuration;
-import org.apache.flink.runtime.blob.BlobClient;
-import org.apache.flink.runtime.blob.PermanentBlobKey;
+import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.client.JobStatusMessage;
import org.apache.flink.runtime.client.JobSubmissionException;
import org.apache.flink.runtime.clusterframework.messages.GetClusterStatusResponse;
@@ -42,6 +43,7 @@
import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException;
import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.rest.FileUpload;
import org.apache.flink.runtime.rest.RestClient;
import org.apache.flink.runtime.rest.handler.async.AsynchronousOperationInfo;
import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
@@ -49,8 +51,6 @@
import org.apache.flink.runtime.rest.handler.job.rescaling.RescalingStatusMessageParameters;
import org.apache.flink.runtime.rest.handler.job.rescaling.RescalingTriggerHeaders;
import org.apache.flink.runtime.rest.handler.job.rescaling.RescalingTriggerMessageParameters;
-import org.apache.flink.runtime.rest.messages.BlobServerPortHeaders;
-import org.apache.flink.runtime.rest.messages.BlobServerPortResponseBody;
import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
import org.apache.flink.runtime.rest.messages.EmptyResponseBody;
@@ -67,6 +67,7 @@
import org.apache.flink.runtime.rest.messages.ResponseBody;
import org.apache.flink.runtime.rest.messages.TerminationModeQueryParameter;
import org.apache.flink.runtime.rest.messages.TriggerId;
+import org.apache.flink.runtime.rest.messages.cluster.ShutdownHeaders;
import org.apache.flink.runtime.rest.messages.job.JobDetailsHeaders;
import org.apache.flink.runtime.rest.messages.job.JobDetailsInfo;
import org.apache.flink.runtime.rest.messages.job.JobExecutionResultHeaders;
@@ -86,10 +87,10 @@
import org.apache.flink.runtime.rest.messages.queue.AsynchronouslyCreatedResource;
import org.apache.flink.runtime.rest.messages.queue.QueueStatus;
import org.apache.flink.runtime.rest.util.RestClientException;
+import org.apache.flink.runtime.rest.util.RestConstants;
import org.apache.flink.runtime.util.ExecutorThreadFactory;
import org.apache.flink.runtime.util.LeaderConnectionInfo;
import org.apache.flink.runtime.util.LeaderRetrievalUtils;
-import org.apache.flink.runtime.util.ScalaUtils;
import org.apache.flink.runtime.webmonitor.retriever.LeaderRetriever;
import org.apache.flink.util.ExceptionUtils;
import org.apache.flink.util.ExecutorUtils;
@@ -99,21 +100,22 @@
import org.apache.flink.util.function.CheckedSupplier;
import org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException;
-
-import akka.actor.AddressFromURIString;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
-import java.net.InetSocketAddress;
+import java.io.ObjectOutputStream;
import java.net.MalformedURLException;
import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.ExecutionException;
@@ -243,7 +245,8 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
try {
return jobSubmissionFuture.get();
} catch (Exception e) {
- throw new ProgramInvocationException("Could not submit job " + jobGraph.getJobID() + '.', ExceptionUtils.stripExecutionException(e));
+ throw new ProgramInvocationException("Could not submit job",
+ jobGraph.getJobID(), ExceptionUtils.stripExecutionException(e));
}
} else {
final CompletableFuture<JobResult> jobResultFuture = jobSubmissionFuture.thenCompose(
@@ -253,16 +256,17 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
try {
jobResult = jobResultFuture.get();
} catch (Exception e) {
- throw new ProgramInvocationException("Could not retrieve the execution result.", ExceptionUtils.stripExecutionException(e));
+ throw new ProgramInvocationException("Could not retrieve the execution result.",
+ jobGraph.getJobID(), ExceptionUtils.stripExecutionException(e));
}
try {
this.lastJobExecutionResult = jobResult.toJobExecutionResult(classLoader);
return lastJobExecutionResult;
} catch (JobResult.WrappedJobException we) {
- throw new ProgramInvocationException(we.getCause());
+ throw new ProgramInvocationException("Job failed.", jobGraph.getJobID(), we.getCause());
} catch (IOException | ClassNotFoundException e) {
- throw new ProgramInvocationException(e);
+ throw new ProgramInvocationException("Job failed.", jobGraph.getJobID(), e);
}
}
}
@@ -273,7 +277,9 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
final JobMessageParameters params = new JobMessageParameters();
params.jobPathParameter.resolve(jobId);
- CompletableFuture<JobDetailsInfo> responseFuture = sendRequest(detailsHeaders, params);
+ CompletableFuture<JobDetailsInfo> responseFuture = sendRequest(
+ detailsHeaders,
+ params);
return responseFuture.thenApply(JobDetailsInfo::getJobStatus);
}
@@ -292,11 +298,9 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
() -> {
final JobMessageParameters messageParameters = new JobMessageParameters();
messageParameters.jobPathParameter.resolve(jobId);
- return sendRetryableRequest(
+ return sendRequest(
JobExecutionResultHeaders.getInstance(),
- messageParameters,
- EmptyRequestBody.getInstance(),
- isConnectionProblemException().or(isHttpStatusUnsuccessfulException()));
+ messageParameters);
});
}
@@ -311,42 +315,61 @@ public JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader)
// we have to enable queued scheduling because slot will be allocated lazily
jobGraph.setAllowQueuedScheduling(true);
- log.info("Requesting blob server port.");
- CompletableFuture<BlobServerPortResponseBody> portFuture = sendRequest(
- BlobServerPortHeaders.getInstance());
-
- CompletableFuture<JobGraph> jobUploadFuture = portFuture.thenCombine(
- getDispatcherAddress(),
- (BlobServerPortResponseBody response, String dispatcherAddress) -> {
- log.info("Uploading jar files.");
- final int blobServerPort = response.port;
- final InetSocketAddress address = new InetSocketAddress(dispatcherAddress, blobServerPort);
- final List<PermanentBlobKey> keys;
- try {
- keys = BlobClient.uploadJarFiles(address, flinkConfig, jobGraph.getJobID(), jobGraph.getUserJars());
- } catch (IOException ioe) {
- throw new CompletionException(new FlinkException("Could not upload job jar files.", ioe));
+ CompletableFuture<java.nio.file.Path> jobGraphFileFuture = CompletableFuture.supplyAsync(() -> {
+ try {
+ final java.nio.file.Path jobGraphFile = Files.createTempFile("flink-jobgraph", ".bin");
+ try (ObjectOutputStream objectOut = new ObjectOutputStream(Files.newOutputStream(jobGraphFile))) {
+ objectOut.writeObject(jobGraph);
}
+ return jobGraphFile;
+ } catch (IOException e) {
+ throw new CompletionException(new FlinkException("Failed to serialize JobGraph.", e));
+ }
+ }, executorService);
- for (PermanentBlobKey key : keys) {
- jobGraph.addBlob(key);
- }
+ CompletableFuture<Tuple2<JobSubmitRequestBody, Collection<FileUpload>>> requestFuture = jobGraphFileFuture.thenApply(jobGraphFile -> {
+ List<String> jarFileNames = new ArrayList<>(8);
+ List<JobSubmitRequestBody.DistributedCacheFile> artifactFileNames = new ArrayList<>(8);
+ Collection<FileUpload> filesToUpload = new ArrayList<>(8);
- return jobGraph;
- });
+ filesToUpload.add(new FileUpload(jobGraphFile, RestConstants.CONTENT_TYPE_BINARY));
- CompletableFuture<JobSubmitResponseBody> submissionFuture = jobUploadFuture.thenCompose(
- (JobGraph jobGraphToSubmit) -> {
- log.info("Submitting job graph.");
+ for (Path jar : jobGraph.getUserJars()) {
+ jarFileNames.add(jar.getName());
+ filesToUpload.add(new FileUpload(Paths.get(jar.toUri()), RestConstants.CONTENT_TYPE_JAR));
+ }
- try {
- return sendRequest(
- JobSubmitHeaders.getInstance(),
- new JobSubmitRequestBody(jobGraph));
- } catch (IOException ioe) {
- throw new CompletionException(new FlinkException("Could not create JobSubmitRequestBody.", ioe));
- }
- });
+ for (Map.Entry<String, DistributedCache.DistributedCacheEntry> artifacts : jobGraph.getUserArtifacts().entrySet()) {
+ artifactFileNames.add(new JobSubmitRequestBody.DistributedCacheFile(artifacts.getKey(), new Path(artifacts.getValue().filePath).getName()));
+ filesToUpload.add(new FileUpload(Paths.get(artifacts.getValue().filePath), RestConstants.CONTENT_TYPE_BINARY));
+ }
+
+ final JobSubmitRequestBody requestBody = new JobSubmitRequestBody(
+ jobGraphFile.getFileName().toString(),
+ jarFileNames,
+ artifactFileNames);
+
+ return Tuple2.of(requestBody, Collections.unmodifiableCollection(filesToUpload));
+ });
+
+ final CompletableFuture<JobSubmitResponseBody> submissionFuture = requestFuture.thenCompose(
+ requestAndFileUploads -> sendRetriableRequest(
+ JobSubmitHeaders.getInstance(),
+ EmptyMessageParameters.getInstance(),
+ requestAndFileUploads.f0,
+ requestAndFileUploads.f1,
+ isConnectionProblemOrServiceUnavailable())
+ );
+
+ submissionFuture
+ .thenCombine(jobGraphFileFuture, (ignored, jobGraphFile) -> jobGraphFile)
+ .thenAccept(jobGraphFile -> {
+ try {
+ Files.delete(jobGraphFile);
+ } catch (IOException e) {
+ log.warn("Could not delete temporary file {}.", jobGraphFile, e);
+ }
+ });
return submissionFuture
.thenApply(
@@ -364,8 +387,7 @@ public void stop(JobID jobID) throws Exception {
params.terminationModeQueryParameter.resolve(Collections.singletonList(TerminationModeQueryParameter.TerminationMode.STOP));
CompletableFuture<EmptyResponseBody> responseFuture = sendRequest(
JobTerminationHeaders.getInstance(),
- params
- );
+ params);
responseFuture.get(timeout.toMillis(), TimeUnit.MILLISECONDS);
}
@@ -376,8 +398,7 @@ public void cancel(JobID jobID) throws Exception {
params.terminationModeQueryParameter.resolve(Collections.singletonList(TerminationModeQueryParameter.TerminationMode.CANCEL));
CompletableFuture<EmptyResponseBody> responseFuture = sendRequest(
JobTerminationHeaders.getInstance(),
- params
- );
+ params);
responseFuture.get(timeout.toMillis(), TimeUnit.MILLISECONDS);
}
@@ -389,7 +410,7 @@ public String cancelWithSavepoint(JobID jobId, @Nullable String savepointDirecto
@Override
public CompletableFuture<String> triggerSavepoint(
final JobID jobId,
- final @Nullable String savepointDirectory) throws FlinkException {
+ final @Nullable String savepointDirectory) {
return triggerSavepoint(jobId, savepointDirectory, false);
}
@@ -427,8 +448,7 @@ public String cancelWithSavepoint(JobID jobId, @Nullable String savepointDirecto
CompletableFuture<JobAccumulatorsInfo> responseFuture = sendRequest(
accumulatorsHeaders,
- accMsgParams
- );
+ accMsgParams);
Map<String, OptionalFailure<Object>> result = Collections.emptyMap();
@@ -461,16 +481,14 @@ public String cancelWithSavepoint(JobID jobId, @Nullable String savepointDirecto
savepointStatusHeaders.getUnresolvedMessageParameters();
savepointStatusMessageParameters.jobIdPathParameter.resolve(jobId);
savepointStatusMessageParameters.triggerIdPathParameter.resolve(triggerID);
- return sendRetryableRequest(
+ return sendRequest(
savepointStatusHeaders,
- savepointStatusMessageParameters,
- EmptyRequestBody.getInstance(),
- isConnectionProblemException());
+ savepointStatusMessageParameters);
});
}
@Override
- public CompletableFuture<Collection<JobStatusMessage>> listJobs() throws Exception {
+ public CompletableFuture<Collection<JobStatusMessage>> listJobs() {
return sendRequest(JobsOverviewHeaders.getInstance())
.thenApply(
(multipleJobsDetails) -> multipleJobsDetails
@@ -506,8 +524,7 @@ public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalExc
final CompletableFuture<TriggerResponse> rescalingTriggerResponseFuture = sendRequest(
rescalingTriggerHeaders,
- rescalingTriggerMessageParameters,
- EmptyRequestBody.getInstance());
+ rescalingTriggerMessageParameters);
final CompletableFuture<AsynchronousOperationInfo> rescalingOperationFuture = rescalingTriggerResponseFuture.thenCompose(
(TriggerResponse triggerResponse) -> {
@@ -519,11 +536,9 @@ public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalExc
rescalingStatusMessageParameters.triggerIdPathParameter.resolve(triggerId);
return pollResourceAsync(
- () -> sendRetryableRequest(
+ () -> sendRequest(
rescalingStatusHeaders,
- rescalingStatusMessageParameters,
- EmptyRequestBody.getInstance(),
- isConnectionProblemException()));
+ rescalingStatusMessageParameters));
});
return rescalingOperationFuture.thenApply(
@@ -542,7 +557,6 @@ public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalExc
final CompletableFuture<TriggerResponse> savepointDisposalTriggerFuture = sendRequest(
SavepointDisposalTriggerHeaders.getInstance(),
- EmptyMessageParameters.getInstance(),
savepointDisposalRequest);
final CompletableFuture<AsynchronousOperationInfo> savepointDisposalFuture = savepointDisposalTriggerFuture.thenCompose(
@@ -553,11 +567,9 @@ public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalExc
savepointDisposalStatusMessageParameters.triggerIdPathParameter.resolve(triggerId);
return pollResourceAsync(
- () -> sendRetryableRequest(
+ () -> sendRequest(
savepointDisposalStatusHeaders,
- savepointDisposalStatusMessageParameters,
- EmptyRequestBody.getInstance(),
- isConnectionProblemException()));
+ savepointDisposalStatusMessageParameters));
});
return savepointDisposalFuture.thenApply(
@@ -570,6 +582,17 @@ public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalExc
});
}
+ @Override
+ public void shutDownCluster() {
+ try {
+ sendRequest(ShutdownHeaders.getInstance()).get();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (ExecutionException e) {
+ log.error("Error while shutting down cluster", e);
+ }
+ }
+
/**
* Creates a {@code CompletableFuture} that polls a {@code AsynchronouslyCreatedResource} until
* its {@link AsynchronouslyCreatedResource#queueStatus() QueueStatus} becomes
@@ -664,27 +687,28 @@ public int getMaxSlots() {
return sendRequest(messageHeaders, EmptyMessageParameters.getInstance(), request);
}
- private <M extends MessageHeaders<EmptyRequestBody, P, EmptyMessageParameters>, P extends ResponseBody> CompletableFuture<P>
+ @VisibleForTesting
+ <M extends MessageHeaders<EmptyRequestBody, P, EmptyMessageParameters>, P extends ResponseBody> CompletableFuture<P>
sendRequest(M messageHeaders) {
return sendRequest(messageHeaders, EmptyMessageParameters.getInstance(), EmptyRequestBody.getInstance());
}
private <M extends MessageHeaders<R, P, U>, U extends MessageParameters, R extends RequestBody, P extends ResponseBody> CompletableFuture<P>
sendRequest(M messageHeaders, U messageParameters, R request) {
- return getWebMonitorBaseUrl().thenCompose(webMonitorBaseUrl -> {
- try {
- return restClient.sendRequest(webMonitorBaseUrl.getHost(), webMonitorBaseUrl.getPort(), messageHeaders, messageParameters, request);
- } catch (IOException e) {
- throw new CompletionException(e);
- }
- });
+ return sendRetriableRequest(
+ messageHeaders, messageParameters, request, isConnectionProblemOrServiceUnavailable());
}
private <M extends MessageHeaders<R, P, U>, U extends MessageParameters, R extends RequestBody, P extends ResponseBody> CompletableFuture<P>
- sendRetryableRequest(M messageHeaders, U messageParameters, R request, Predicate<Throwable> retryPredicate) {
+ sendRetriableRequest(M messageHeaders, U messageParameters, R request, Predicate<Throwable> retryPredicate) {
+ return sendRetriableRequest(messageHeaders, messageParameters, request, Collections.emptyList(), retryPredicate);
+ }
+
+ private <M extends MessageHeaders<R, P, U>, U extends MessageParameters, R extends RequestBody, P extends ResponseBody> CompletableFuture<P>
+ sendRetriableRequest(M messageHeaders, U messageParameters, R request, Collection<FileUpload> filesToUpload, Predicate<Throwable> retryPredicate) {
return retry(() -> getWebMonitorBaseUrl().thenCompose(webMonitorBaseUrl -> {
try {
- return restClient.sendRequest(webMonitorBaseUrl.getHost(), webMonitorBaseUrl.getPort(), messageHeaders, messageParameters, request);
+ return restClient.sendRequest(webMonitorBaseUrl.getHost(), webMonitorBaseUrl.getPort(), messageHeaders, messageParameters, request, filesToUpload);
} catch (IOException e) {
throw new CompletionException(e);
}
@@ -702,6 +726,10 @@ public int getMaxSlots() {
new ScheduledExecutorServiceAdapter(retryExecutorService));
}
+ private static Predicate<Throwable> isConnectionProblemOrServiceUnavailable() {
+ return isConnectionProblemException().or(isServiceUnavailable());
+ }
+
private static Predicate<Throwable> isConnectionProblemException() {
return (throwable) ->
ExceptionUtils.findThrowable(throwable, java.net.ConnectException.class).isPresent() ||
@@ -710,13 +738,17 @@ public int getMaxSlots() {
ExceptionUtils.findThrowable(throwable, IOException.class).isPresent();
}
- private static Predicate<Throwable> isHttpStatusUnsuccessfulException() {
+ private static Predicate<Throwable> isServiceUnavailable() {
+ return httpExceptionCodePredicate(code -> code == HttpResponseStatus.SERVICE_UNAVAILABLE.code());
+ }
+
+ private static Predicate<Throwable> httpExceptionCodePredicate(Predicate<Integer> statusCodePredicate) {
return (throwable) -> ExceptionUtils.findThrowable(throwable, RestClientException.class)
- .map(restClientException -> {
- final int code = restClientException.getHttpResponseStatus().code();
- return code < 200 || code > 299;
- })
- .orElse(false);
+ .map(restClientException -> {
+ final int code = restClientException.getHttpResponseStatus().code();
+ return statusCodePredicate.test(code);
+ })
+ .orElse(false);
}
@VisibleForTesting
@@ -734,26 +766,4 @@ public int getMaxSlots() {
}
}, executorService);
}
-
- private CompletableFuture<String> getDispatcherAddress() {
- return FutureUtils.orTimeout(
- dispatcherLeaderRetriever.getLeaderFuture(),
- restClusterClientConfiguration.getAwaitLeaderTimeout(),
- TimeUnit.MILLISECONDS)
- .thenApplyAsync(leaderAddressSessionId -> {
- final String address = leaderAddressSessionId.f0;
- final Optional<String> host = ScalaUtils.<String>toJava(AddressFromURIString.parse(address).host());
-
- return host.orElseGet(() -> {
- // if the dispatcher address does not contain a host part, then assume it's running
- // on the same machine as the client
- log.info("The dispatcher seems to run without remoting enabled. This indicates that we are " +
- "in a test. This can only work if the RestClusterClient runs on the same machine. " +
- "Assuming, therefore, 'localhost' as the host.");
-
- return "localhost";
- });
- }, executorService);
- }
-
}
diff --git a/flink-clients/src/test/java/org/apache/flink/client/cli/CliFrontendListTest.java b/flink-clients/src/test/java/org/apache/flink/client/cli/CliFrontendListTest.java
index a8a7f006bda..be37cccdd77 100644
--- a/flink-clients/src/test/java/org/apache/flink/client/cli/CliFrontendListTest.java
+++ b/flink-clients/src/test/java/org/apache/flink/client/cli/CliFrontendListTest.java
@@ -18,18 +18,24 @@
package org.apache.flink.client.cli;
+import org.apache.flink.api.common.JobID;
import org.apache.flink.client.cli.util.MockedCliFrontend;
import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.runtime.jobgraph.JobStatus;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.mockito.Mockito;
+import java.util.Arrays;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.when;
@@ -50,15 +56,35 @@ public static void shutdown() {
}
@Test
- public void testList() throws Exception {
- // test list properly
+ public void testListOptions() throws Exception {
+ // test configure all job
{
- String[] parameters = {"-r", "-s"};
- ClusterClient<String> clusterClient = createClusterClient();
- MockedCliFrontend testFrontend = new MockedCliFrontend(clusterClient);
- testFrontend.list(parameters);
- Mockito.verify(clusterClient, times(1))
- .listJobs();
+ String[] parameters = {"-a"};
+ ListOptions options = new ListOptions(CliFrontendParser.parse(
+ CliFrontendParser.getListCommandOptions(), parameters, true));
+ assertTrue(options.showAll());
+ assertFalse(options.showRunning());
+ assertFalse(options.showScheduled());
+ }
+
+ // test configure running job
+ {
+ String[] parameters = {"-r"};
+ ListOptions options = new ListOptions(CliFrontendParser.parse(
+ CliFrontendParser.getListCommandOptions(), parameters, true));
+ assertFalse(options.showAll());
+ assertTrue(options.showRunning());
+ assertFalse(options.showScheduled());
+ }
+
+ // test configure scheduled job
+ {
+ String[] parameters = {"-s"};
+ ListOptions options = new ListOptions(CliFrontendParser.parse(
+ CliFrontendParser.getListCommandOptions(), parameters, true));
+ assertFalse(options.showAll());
+ assertFalse(options.showRunning());
+ assertTrue(options.showScheduled());
}
}
@@ -72,11 +98,28 @@ public void testUnrecognizedOption() throws Exception {
testFrontend.list(parameters);
}
+ @Test
+ public void testList() throws Exception {
+ // test list properly
+ {
+ String[] parameters = {"-r", "-s", "-a"};
+ ClusterClient<String> clusterClient = createClusterClient();
+ MockedCliFrontend testFrontend = new MockedCliFrontend(clusterClient);
+ testFrontend.list(parameters);
+ Mockito.verify(clusterClient, times(1))
+ .listJobs();
+ }
+ }
+
private static ClusterClient<String> createClusterClient() throws Exception {
final ClusterClient<String> clusterClient = mock(ClusterClient.class);
-
- when(clusterClient.listJobs()).thenReturn(CompletableFuture.completedFuture(Collections.emptyList()));
-
+ when(clusterClient.listJobs()).thenReturn(CompletableFuture.completedFuture(Arrays.asList(
+ new JobStatusMessage(new JobID(), "job1", JobStatus.RUNNING, 1L),
+ new JobStatusMessage(new JobID(), "job2", JobStatus.CREATED, 1L),
+ new JobStatusMessage(new JobID(), "job3", JobStatus.SUSPENDING, 3L),
+ new JobStatusMessage(new JobID(), "job4", JobStatus.SUSPENDING, 2L),
+ new JobStatusMessage(new JobID(), "job5", JobStatus.FINISHED, 3L)
+ )));
return clusterClient;
}
}
diff --git a/flink-clients/src/test/java/org/apache/flink/client/cli/util/DummyClusterDescriptor.java b/flink-clients/src/test/java/org/apache/flink/client/cli/util/DummyClusterDescriptor.java
index df2f3f78591..7620ae22157 100644
--- a/flink-clients/src/test/java/org/apache/flink/client/cli/util/DummyClusterDescriptor.java
+++ b/flink-clients/src/test/java/org/apache/flink/client/cli/util/DummyClusterDescriptor.java
@@ -60,7 +60,7 @@ public String getClusterDescription() {
}
@Override
- public void terminateCluster(T clusterId) throws FlinkException {
+ public void killCluster(T clusterId) throws FlinkException {
throw new UnsupportedOperationException("Cannot terminate a dummy cluster.");
}
diff --git a/flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramTest.java b/flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramTest.java
index e137adae3d3..355e663774e 100644
--- a/flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramTest.java
+++ b/flink-clients/src/test/java/org/apache/flink/client/program/PackagedProgramTest.java
@@ -19,18 +19,29 @@
package org.apache.flink.client.program;
import org.apache.flink.client.cli.CliFrontendTestUtils;
+import org.apache.flink.configuration.ConfigConstants;
import org.junit.Assert;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
import java.io.File;
+import java.io.FileOutputStream;
import java.io.PrintStream;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
/**
* Tests for the {@link PackagedProgramTest}.
*/
public class PackagedProgramTest {
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
@Test
public void testGetPreviewPlan() {
try {
@@ -56,6 +67,23 @@ public void testGetPreviewPlan() {
}
}
+ @Test
+ public void testExtractContainedLibraries() throws Exception {
+ String s = "testExtractContainedLibraries";
+ byte[] nestedJarContent = s.getBytes(ConfigConstants.DEFAULT_CHARSET);
+ File fakeJar = temporaryFolder.newFile("test.jar");
+ try (ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(fakeJar))) {
+ ZipEntry entry = new ZipEntry("lib/internalTest.jar");
+ zos.putNextEntry(entry);
+ zos.write(nestedJarContent);
+ zos.closeEntry();
+ }
+
+ final List<File> files = PackagedProgram.extractContainedLibraries(fakeJar.toURI().toURL());
+ Assert.assertEquals(1, files.size());
+ Assert.assertArrayEquals(nestedJarContent, Files.readAllBytes(files.iterator().next().toPath()));
+ }
+
private static final class NullOutputStream extends java.io.OutputStream {
@Override
public void write(int b) {}
diff --git a/flink-clients/src/test/java/org/apache/flink/client/program/rest/RestClusterClientTest.java b/flink-clients/src/test/java/org/apache/flink/client/program/rest/RestClusterClientTest.java
index fd05cadbec2..75f16c03330 100644
--- a/flink-clients/src/test/java/org/apache/flink/client/program/rest/RestClusterClientTest.java
+++ b/flink-clients/src/test/java/org/apache/flink/client/program/rest/RestClusterClientTest.java
@@ -40,6 +40,7 @@
import org.apache.flink.runtime.messages.Acknowledge;
import org.apache.flink.runtime.messages.webmonitor.JobDetails;
import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.rest.HttpMethodWrapper;
import org.apache.flink.runtime.rest.RestClient;
import org.apache.flink.runtime.rest.RestClientConfiguration;
import org.apache.flink.runtime.rest.RestServerEndpoint;
@@ -52,8 +53,6 @@
import org.apache.flink.runtime.rest.handler.async.AsynchronousOperationResult;
import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
import org.apache.flink.runtime.rest.messages.AccumulatorsIncludeSerializedValueQueryParameter;
-import org.apache.flink.runtime.rest.messages.BlobServerPortHeaders;
-import org.apache.flink.runtime.rest.messages.BlobServerPortResponseBody;
import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
import org.apache.flink.runtime.rest.messages.EmptyResponseBody;
@@ -90,7 +89,6 @@
import org.apache.flink.runtime.rpc.RpcUtils;
import org.apache.flink.runtime.util.ExecutorThreadFactory;
import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
-import org.apache.flink.testutils.category.New;
import org.apache.flink.util.ExceptionUtils;
import org.apache.flink.util.FlinkException;
import org.apache.flink.util.OptionalFailure;
@@ -107,7 +105,6 @@
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import org.junit.experimental.categories.Category;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;
@@ -152,7 +149,6 @@
* <p>These tests verify that the client uses the appropriate headers for each
* request, properly constructs the request bodies/parameters and processes the responses correctly.
*/
-@Category(New.class)
public class RestClusterClientTest extends TestLogger {
private static final String REST_ADDRESS = "http://localhost:1234";
@@ -228,7 +224,6 @@ public void tearDown() throws Exception {
@Test
public void testJobSubmitCancelStop() throws Exception {
- TestBlobServerPortHandler portHandler = new TestBlobServerPortHandler();
TestJobSubmitHandler submitHandler = new TestJobSubmitHandler();
TestJobTerminationHandler terminationHandler = new TestJobTerminationHandler();
TestJobExecutionResultHandler testJobExecutionResultHandler =
@@ -239,15 +234,12 @@ public void testJobSubmitCancelStop() throws Exception {
.build()));
try (TestRestServerEndpoint ignored = createRestServerEndpoint(
- portHandler,
submitHandler,
terminationHandler,
testJobExecutionResultHandler)) {
- Assert.assertFalse(portHandler.portRetrieved);
Assert.assertFalse(submitHandler.jobSubmitted);
restClusterClient.submitJob(jobGraph, ClassLoader.getSystemClassLoader());
- Assert.assertTrue(portHandler.portRetrieved);
Assert.assertTrue(submitHandler.jobSubmitted);
Assert.assertFalse(terminationHandler.jobCanceled);
@@ -266,11 +258,9 @@ public void testJobSubmitCancelStop() throws Exception {
@Test
public void testDetachedJobSubmission() throws Exception {
- final TestBlobServerPortHandler testBlobServerPortHandler = new TestBlobServerPortHandler();
final TestJobSubmitHandler testJobSubmitHandler = new TestJobSubmitHandler();
try (TestRestServerEndpoint ignored = createRestServerEndpoint(
- testBlobServerPortHandler,
testJobSubmitHandler)) {
restClusterClient.setDetached(true);
@@ -284,20 +274,6 @@ public void testDetachedJobSubmission() throws Exception {
}
- private class TestBlobServerPortHandler extends TestHandler<EmptyRequestBody, BlobServerPortResponseBody, EmptyMessageParameters> {
- private volatile boolean portRetrieved = false;
-
- private TestBlobServerPortHandler() {
- super(BlobServerPortHeaders.getInstance());
- }
-
- @Override
- protected CompletableFuture<BlobServerPortResponseBody> handleRequest(@Nonnull HandlerRequest<EmptyRequestBody, EmptyMessageParameters> request, @Nonnull DispatcherGateway gateway) throws RestHandlerException {
- portRetrieved = true;
- return CompletableFuture.completedFuture(new BlobServerPortResponseBody(12000));
- }
- }
-
private class TestJobSubmitHandler extends TestHandler<JobSubmitRequestBody, JobSubmitResponseBody, EmptyMessageParameters> {
private volatile boolean jobSubmitted = false;
@@ -372,7 +348,7 @@ private TestJobExecutionResultHandler(
public void testSubmitJobAndWaitForExecutionResult() throws Exception {
final TestJobExecutionResultHandler testJobExecutionResultHandler =
new TestJobExecutionResultHandler(
- new RestHandlerException("should trigger retry", HttpResponseStatus.NOT_FOUND),
+ new RestHandlerException("should trigger retry", HttpResponseStatus.SERVICE_UNAVAILABLE),
JobExecutionResultResponseBody.inProgress(),
JobExecutionResultResponseBody.created(new JobResult.Builder()
.jobId(jobId)
@@ -392,7 +368,6 @@ public void testSubmitJobAndWaitForExecutionResult() throws Exception {
try (TestRestServerEndpoint ignored = createRestServerEndpoint(
testJobExecutionResultHandler,
- new TestBlobServerPortHandler(),
new TestJobSubmitHandler())) {
JobExecutionResult jobExecutionResult;
@@ -712,6 +687,104 @@ public void testRESTManualConfigurationOverride() throws Exception {
assertThat(webMonitorBaseUrl.getPort(), equalTo(manualPort));
}
+ /**
+ * Tests that the send operation is being retried.
+ */
+ @Test
+ public void testRetriableSendOperationIfConnectionErrorOrServiceUnavailable() throws Exception {
+ final PingRestHandler pingRestHandler = new PingRestHandler(
+ FutureUtils.completedExceptionally(new RestHandlerException("test exception", HttpResponseStatus.SERVICE_UNAVAILABLE)),
+ CompletableFuture.completedFuture(EmptyResponseBody.getInstance()));
+
+ try (final TestRestServerEndpoint restServerEndpoint = createRestServerEndpoint(pingRestHandler)) {
+ final AtomicBoolean firstPollFailed = new AtomicBoolean();
+ failHttpRequest = (messageHeaders, messageParameters, requestBody) ->
+ messageHeaders instanceof PingRestHandlerHeaders && !firstPollFailed.getAndSet(true);
+
+ restClusterClient.sendRequest(PingRestHandlerHeaders.INSTANCE).get();
+ }
+ }
+
+ /**
+ * Tests that the send operation is not being retried when receiving a NOT_FOUND return code.
+ */
+ @Test
+ public void testSendIsNotRetriableIfHttpNotFound() throws Exception {
+ final String exceptionMessage = "test exception";
+ final PingRestHandler pingRestHandler = new PingRestHandler(
+ FutureUtils.completedExceptionally(new RestHandlerException(exceptionMessage, HttpResponseStatus.NOT_FOUND)));
+
+ try (final TestRestServerEndpoint restServerEndpoint = createRestServerEndpoint(pingRestHandler)) {
+ try {
+ restClusterClient.sendRequest(PingRestHandlerHeaders.INSTANCE).get();
+ fail("The rest request should have failed.");
+ } catch (Exception e) {
+ assertThat(ExceptionUtils.findThrowableWithMessage(e, exceptionMessage).isPresent(), is(true));
+ }
+ }
+ }
+
+ private class PingRestHandler extends TestHandler<EmptyRequestBody, EmptyResponseBody, EmptyMessageParameters> {
+
+ private final Queue<CompletableFuture<EmptyResponseBody>> responseQueue;
+
+ private PingRestHandler(CompletableFuture<EmptyResponseBody> ... responses) {
+ super(PingRestHandlerHeaders.INSTANCE);
+ responseQueue = new ArrayDeque<>(Arrays.asList(responses));
+ }
+
+ @Override
+ protected CompletableFuture<EmptyResponseBody> handleRequest(@Nonnull HandlerRequest<EmptyRequestBody, EmptyMessageParameters> request, @Nonnull DispatcherGateway gateway) throws RestHandlerException {
+ final CompletableFuture<EmptyResponseBody> result = responseQueue.poll();
+
+ if (result != null) {
+ return result;
+ } else {
+ return CompletableFuture.completedFuture(EmptyResponseBody.getInstance());
+ }
+ }
+ }
+
+ private static final class PingRestHandlerHeaders implements MessageHeaders<EmptyRequestBody, EmptyResponseBody, EmptyMessageParameters> {
+
+ static final PingRestHandlerHeaders INSTANCE = new PingRestHandlerHeaders();
+
+ @Override
+ public Class<EmptyResponseBody> getResponseClass() {
+ return EmptyResponseBody.class;
+ }
+
+ @Override
+ public HttpResponseStatus getResponseStatusCode() {
+ return HttpResponseStatus.OK;
+ }
+
+ @Override
+ public String getDescription() {
+ return "foobar";
+ }
+
+ @Override
+ public Class<EmptyRequestBody> getRequestClass() {
+ return EmptyRequestBody.class;
+ }
+
+ @Override
+ public EmptyMessageParameters getUnresolvedMessageParameters() {
+ return EmptyMessageParameters.getInstance();
+ }
+
+ @Override
+ public HttpMethodWrapper getHttpMethod() {
+ return HttpMethodWrapper.GET;
+ }
+
+ @Override
+ public String getTargetRestEndpointURL() {
+ return "/foobar";
+ }
+ }
+
private class TestAccumulatorHandler extends TestHandler<EmptyRequestBody, JobAccumulatorsInfo, JobAccumulatorsMessageParameters> {
public TestAccumulatorHandler() {
diff --git a/flink-connectors/flink-connector-cassandra/pom.xml b/flink-connectors/flink-connector-cassandra/pom.xml
index d1126233420..710d3f33f22 100644
--- a/flink-connectors/flink-connector-cassandra/pom.xml
+++ b/flink-connectors/flink-connector-cassandra/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraPojoSink.java b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraPojoSink.java
index e060ce08330..da70da17472 100644
--- a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraPojoSink.java
+++ b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraPojoSink.java
@@ -20,6 +20,7 @@
import org.apache.flink.configuration.Configuration;
import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
import com.datastax.driver.mapping.Mapper;
import com.datastax.driver.mapping.MappingManager;
import com.google.common.util.concurrent.ListenableFuture;
@@ -41,6 +42,7 @@
protected final Class<IN> clazz;
private final MapperOptions options;
+ private final String keyspace;
protected transient Mapper<IN> mapper;
protected transient MappingManager mappingManager;
@@ -50,13 +52,22 @@
* @param clazz Class instance
*/
public CassandraPojoSink(Class<IN> clazz, ClusterBuilder builder) {
- this(clazz, builder, null);
+ this(clazz, builder, null, null);
}
public CassandraPojoSink(Class<IN> clazz, ClusterBuilder builder, @Nullable MapperOptions options) {
+ this(clazz, builder, options, null);
+ }
+
+ public CassandraPojoSink(Class<IN> clazz, ClusterBuilder builder, String keyspace) {
+ this(clazz, builder, null, keyspace);
+ }
+
+ public CassandraPojoSink(Class<IN> clazz, ClusterBuilder builder, @Nullable MapperOptions options, String keyspace) {
super(builder);
this.clazz = clazz;
this.options = options;
+ this.keyspace = keyspace;
}
@Override
@@ -76,6 +87,11 @@ public void open(Configuration configuration) {
}
}
+ @Override
+ protected Session createSession() {
+ return cluster.connect(keyspace);
+ }
+
@Override
public ListenableFuture<ResultSet> send(IN value) {
return session.executeAsync(mapper.saveQuery(value));
diff --git a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java
index 3543378c3ef..5d5e9efc36e 100644
--- a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java
+++ b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java
@@ -236,6 +236,7 @@ private CassandraSink(SingleOutputStreamOperator<IN> sink) {
protected final TypeSerializer<IN> serializer;
protected final TypeInformation<IN> typeInfo;
protected ClusterBuilder builder;
+ protected String keyspace;
protected MapperOptions mapperOptions;
protected String query;
protected CheckpointCommitter committer;
@@ -258,6 +259,17 @@ public CassandraSinkBuilder(DataStream<IN> input, TypeInformation<IN> typeInfo,
return this;
}
+ /**
+ * Sets the keyspace to be used.
+ *
+ * @param keyspace keyspace to use
+ * @return this builder
+ */
+ public CassandraSinkBuilder<IN> setDefaultKeyspace(String keyspace) {
+ this.keyspace = keyspace;
+ return this;
+ }
+
/**
* Sets the cassandra host to connect to.
*
@@ -381,6 +393,9 @@ protected void sanityCheck() {
if (query == null || query.length() == 0) {
throw new IllegalArgumentException("Query must not be null or empty.");
}
+ if (keyspace != null) {
+ throw new IllegalArgumentException("Specifying a default keyspace is only allowed when using a Pojo-Stream as input.");
+ }
}
@Override
@@ -410,6 +425,9 @@ protected void sanityCheck() {
if (query == null || query.length() == 0) {
throw new IllegalArgumentException("Query must not be null or empty.");
}
+ if (keyspace != null) {
+ throw new IllegalArgumentException("Specifying a default keyspace is only allowed when using a Pojo-Stream as input.");
+ }
}
@Override
@@ -445,7 +463,7 @@ protected void sanityCheck() {
@Override
public CassandraSink<IN> createSink() throws Exception {
- return new CassandraSink<>(input.addSink(new CassandraPojoSink<>(typeInfo.getTypeClass(), builder, mapperOptions)).name("Cassandra Sink"));
+ return new CassandraSink<>(input.addSink(new CassandraPojoSink<>(typeInfo.getTypeClass(), builder, mapperOptions, keyspace)).name("Cassandra Sink"));
}
@Override
@@ -470,6 +488,9 @@ protected void sanityCheck() {
if (query == null || query.length() == 0) {
throw new IllegalArgumentException("Query must not be null or empty.");
}
+ if (keyspace != null) {
+ throw new IllegalArgumentException("Specifying a default keyspace is only allowed when using a Pojo-Stream as input.");
+ }
}
@Override
diff --git a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSinkBase.java b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSinkBase.java
index 6d1b095b0f7..d2ba8e53913 100644
--- a/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSinkBase.java
+++ b/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSinkBase.java
@@ -85,7 +85,11 @@ public void onFailure(Throwable t) {
}
};
this.cluster = builder.getCluster();
- this.session = cluster.connect();
+ this.session = createSession();
+ }
+
+ protected Session createSession() {
+ return cluster.connect();
}
@Override
diff --git a/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java
index 4444fa6fd16..cb5767a0173 100644
--- a/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java
+++ b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java
@@ -429,6 +429,25 @@ public void testCassandraPojoAtLeastOnceSink() throws Exception {
Assert.assertEquals(20, rs.all().size());
}
+ @Test
+ public void testCassandraPojoNoAnnotatedKeyspaceAtLeastOnceSink() throws Exception {
+ session.execute(CREATE_TABLE_QUERY.replace(TABLE_NAME_VARIABLE, "testPojoNoAnnotatedKeyspace"));
+
+ CassandraPojoSink<PojoNoAnnotatedKeyspace> sink = new CassandraPojoSink<>(PojoNoAnnotatedKeyspace.class, builder, "flink");
+
+ Configuration configuration = new Configuration();
+ sink.open(configuration);
+
+ for (int x = 0; x < 20; x++) {
+ sink.send(new PojoNoAnnotatedKeyspace(UUID.randomUUID().toString(), x, 0));
+ }
+
+ sink.close();
+
+ ResultSet rs = session.execute(SELECT_DATA_QUERY.replace(TABLE_NAME_VARIABLE, "testPojoNoAnnotatedKeyspace"));
+ Assert.assertEquals(20, rs.all().size());
+ }
+
@Test
public void testCassandraTableSink() throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
diff --git a/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/PojoNoAnnotatedKeyspace.java b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/PojoNoAnnotatedKeyspace.java
new file mode 100644
index 00000000000..ab8844d588b
--- /dev/null
+++ b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/PojoNoAnnotatedKeyspace.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.cassandra;
+
+import com.datastax.driver.mapping.annotations.Column;
+import com.datastax.driver.mapping.annotations.Table;
+
+import java.io.Serializable;
+
+/**
+ * Test Pojo with DataStax annotations used (no keyspace).
+ */
+@Table(name = "testPojoNoAnnotatedKeyspace")
+public class PojoNoAnnotatedKeyspace implements Serializable {
+
+ private static final long serialVersionUID = 1038054554690916991L;
+
+ @Column(name = "id")
+ private String id;
+ @Column(name = "counter")
+ private int counter;
+ @Column(name = "batch_id")
+ private int batchID;
+
+ public PojoNoAnnotatedKeyspace(String id, int counter, int batchID) {
+ this.id = id;
+ this.counter = counter;
+ this.batchID = batchID;
+ }
+
+ public String getId() {
+ return id;
+ }
+
+ public void setId(String id) {
+ this.id = id;
+ }
+
+ public int getCounter() {
+ return counter;
+ }
+
+ public void setCounter(int counter) {
+ this.counter = counter;
+ }
+
+ public int getBatchID() {
+ return batchID;
+ }
+
+ public void setBatchID(int batchId) {
+ this.batchID = batchId;
+ }
+}
diff --git a/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/Message.java b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/Message.java
index 512d0ea91d5..930a3cc28c0 100644
--- a/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/Message.java
+++ b/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/Message.java
@@ -33,6 +33,10 @@
@Column(name = "body")
private String message;
+ public Message() {
+ this(null);
+ }
+
public Message(String word) {
this.message = word;
}
diff --git a/flink-connectors/flink-connector-elasticsearch-base/pom.xml b/flink-connectors/flink-connector-elasticsearch-base/pom.xml
index 4b4c93c68ec..abb8fbfecdf 100644
--- a/flink-connectors/flink-connector-elasticsearch-base/pom.xml
+++ b/flink-connectors/flink-connector-elasticsearch-base/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-elasticsearch/pom.xml b/flink-connectors/flink-connector-elasticsearch/pom.xml
index 1714ec37520..044f21bb8f3 100644
--- a/flink-connectors/flink-connector-elasticsearch/pom.xml
+++ b/flink-connectors/flink-connector-elasticsearch/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-elasticsearch/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSink.java b/flink-connectors/flink-connector-elasticsearch/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSink.java
index 9dd8209d750..e8eccd978f4 100644
--- a/flink-connectors/flink-connector-elasticsearch/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSink.java
+++ b/flink-connectors/flink-connector-elasticsearch/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSink.java
@@ -44,7 +44,7 @@
* to come online.
*
* <p>The {@link Map} passed to the constructor is used to create the {@link Node} or {@link TransportClient}. The config
- * keys can be found in the <a href="https://www.elastic.io">Elasticsearch documentation</a>. An important setting is
+ * keys can be found in the <a href="https://www.elastic.co">Elasticsearch documentation</a>. An important setting is
* {@code cluster.name}, which should be set to the name of the cluster that the sink should emit to.
*
* <p>Internally, the sink will use a {@link BulkProcessor} to send {@link ActionRequest ActionRequests}.
diff --git a/flink-connectors/flink-connector-elasticsearch2/pom.xml b/flink-connectors/flink-connector-elasticsearch2/pom.xml
index 1068c1b4826..29fbf3ea338 100644
--- a/flink-connectors/flink-connector-elasticsearch2/pom.xml
+++ b/flink-connectors/flink-connector-elasticsearch2/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-elasticsearch2/src/main/java/org/apache/flink/streaming/connectors/elasticsearch2/ElasticsearchSink.java b/flink-connectors/flink-connector-elasticsearch2/src/main/java/org/apache/flink/streaming/connectors/elasticsearch2/ElasticsearchSink.java
index a17b4d8543f..ffccacf40ac 100644
--- a/flink-connectors/flink-connector-elasticsearch2/src/main/java/org/apache/flink/streaming/connectors/elasticsearch2/ElasticsearchSink.java
+++ b/flink-connectors/flink-connector-elasticsearch2/src/main/java/org/apache/flink/streaming/connectors/elasticsearch2/ElasticsearchSink.java
@@ -38,7 +38,7 @@
* The sink will fail if no cluster can be connected to using the provided transport addresses passed to the constructor.
*
* <p>The {@link Map} passed to the constructor is used to create the {@code TransportClient}. The config keys can be found
- * in the <a href="https://www.elastic.io">Elasticsearch documentation</a>. An important setting is {@code cluster.name},
+ * in the <a href="https://www.elastic.co">Elasticsearch documentation</a>. An important setting is {@code cluster.name},
* which should be set to the name of the cluster that the sink should emit to.
*
* <p>Internally, the sink will use a {@link BulkProcessor} to send {@link ActionRequest ActionRequests}.
diff --git a/flink-connectors/flink-connector-elasticsearch5/pom.xml b/flink-connectors/flink-connector-elasticsearch5/pom.xml
index fc97294a8db..a8a6489f22c 100644
--- a/flink-connectors/flink-connector-elasticsearch5/pom.xml
+++ b/flink-connectors/flink-connector-elasticsearch5/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java b/flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java
index 3307b2c4f2a..6c09337227a 100644
--- a/flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java
+++ b/flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java
@@ -39,7 +39,7 @@
* The sink will fail if no cluster can be connected to using the provided transport addresses passed to the constructor.
*
* <p>The {@link Map} passed to the constructor is used to create the {@code TransportClient}. The config keys can be found
- * in the <a href="https://www.elastic.io">Elasticsearch documentation</a>. An important setting is {@code cluster.name},
+ * in the <a href="https://www.elastic.co">Elasticsearch documentation</a>. An important setting is {@code cluster.name},
* which should be set to the name of the cluster that the sink should emit to.
*
* <p>Internally, the sink will use a {@link BulkProcessor} to send {@link ActionRequest ActionRequests}.
diff --git a/flink-connectors/flink-connector-filesystem/pom.xml b/flink-connectors/flink-connector-filesystem/pom.xml
index 7907242f469..592db4b5106 100644
--- a/flink-connectors/flink-connector-filesystem/pom.xml
+++ b/flink-connectors/flink-connector-filesystem/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
@@ -67,13 +67,6 @@ under the License.
<!-- test dependencies -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-test-utils-junit</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils_${scala.binary.version}</artifactId>
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriter.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriter.java
index 6b2f7d625a1..0f73e8c5c0b 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriter.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriter.java
@@ -40,7 +40,6 @@
import java.io.OutputStream;
import java.util.Arrays;
import java.util.Map;
-import java.util.Objects;
/**
* Implementation of AvroKeyValue writer that can be used in Sink.
@@ -204,7 +203,7 @@ public void setInputType(TypeInformation<?> type, ExecutionConfig executionConfi
}
@Override
- public Writer<Tuple2<K, V>> duplicate() {
+ public AvroKeyValueSinkWriter<K, V> duplicate() {
return new AvroKeyValueSinkWriter<>(this);
}
@@ -335,25 +334,7 @@ public static Schema getSchema(Schema keySchema, Schema valueSchema) {
}
}
- @Override
- public int hashCode() {
- return Objects.hash(super.hashCode(), properties);
- }
-
- @Override
- public boolean equals(Object other) {
- if (this == other) {
- return true;
- }
- if (other == null) {
- return false;
- }
- if (getClass() != other.getClass()) {
- return false;
- }
- AvroKeyValueSinkWriter<K, V> writer = (AvroKeyValueSinkWriter<K, V>) other;
- // field comparison
- return Objects.equals(properties, writer.properties)
- && super.equals(other);
+ Map<String, String> getProperties() {
+ return properties;
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/DateTimeBucketer.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/DateTimeBucketer.java
index 72b482368c5..96c743bbf47 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/DateTimeBucketer.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/DateTimeBucketer.java
@@ -51,7 +51,7 @@
* }</pre>
*
* <p>This will create for example the following bucket path:
- * {@code /base/1976-12-31-14/}
+ * {@code /base/1976-12-31--14/}
*
* @deprecated use {@link org.apache.flink.streaming.connectors.fs.bucketing.DateTimeBucketer} instead.
*/
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
index 709a7c9ffc6..9ec97b7db2c 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
@@ -533,12 +533,9 @@ private Method reflectTruncate(FileSystem fs) {
}
// verify that truncate actually works
- FSDataOutputStream outputStream;
Path testPath = new Path(UUID.randomUUID().toString());
- try {
- outputStream = fs.create(testPath);
+ try (FSDataOutputStream outputStream = fs.create(testPath)) {
outputStream.writeUTF("hello");
- outputStream.close();
} catch (IOException e) {
LOG.error("Could not create file for checking if truncate works.", e);
throw new RuntimeException("Could not create file for checking if truncate works.", e);
@@ -702,9 +699,9 @@ private void handleRestoredBucketState(BucketState bucketState) {
Path validLengthFilePath = getValidLengthPathFor(partPath);
if (!fs.exists(validLengthFilePath) && fs.exists(partPath)) {
LOG.debug("Writing valid-length file for {} to specify valid length {}", partPath, bucketState.currentFileValidLength);
- FSDataOutputStream lengthFileOut = fs.create(validLengthFilePath);
- lengthFileOut.writeUTF(Long.toString(bucketState.currentFileValidLength));
- lengthFileOut.close();
+ try (FSDataOutputStream lengthFileOut = fs.create(validLengthFilePath)) {
+ lengthFileOut.writeUTF(Long.toString(bucketState.currentFileValidLength));
+ }
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriter.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriter.java
index 2f42ef74c52..17b16dd8b46 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriter.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriter.java
@@ -34,7 +34,6 @@
import org.apache.hadoop.io.compress.CompressionCodecFactory;
import java.io.IOException;
-import java.util.Objects;
/**
* A {@link Writer} that writes the bucket files as Hadoop {@link SequenceFile SequenceFiles}.
@@ -152,32 +151,23 @@ public void setInputType(TypeInformation<?> type, ExecutionConfig executionConfi
}
@Override
- public Writer<Tuple2<K, V>> duplicate() {
+ public SequenceFileWriter<K, V> duplicate() {
return new SequenceFileWriter<>(this);
}
- @Override
- public int hashCode() {
- return Objects.hash(super.hashCode(), compressionCodecName, compressionType, keyClass, valueClass);
+ String getCompressionCodecName() {
+ return compressionCodecName;
}
- @Override
- public boolean equals(Object other) {
- if (this == other) {
- return true;
- }
- if (other == null) {
- return false;
- }
- if (getClass() != other.getClass()) {
- return false;
- }
- SequenceFileWriter<K, V> writer = (SequenceFileWriter<K, V>) other;
- // field comparison
- return Objects.equals(compressionCodecName, writer.compressionCodecName)
- && Objects.equals(compressionType, writer.compressionType)
- && Objects.equals(keyClass, writer.keyClass)
- && Objects.equals(valueClass, writer.valueClass)
- && super.equals(other);
+ SequenceFile.CompressionType getCompressionType() {
+ return compressionType;
+ }
+
+ Class<K> getKeyClass() {
+ return keyClass;
+ }
+
+ Class<V> getValueClass() {
+ return valueClass;
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StreamWriterBase.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StreamWriterBase.java
index f625ef38905..d3035a56622 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StreamWriterBase.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StreamWriterBase.java
@@ -23,7 +23,6 @@
import org.apache.hadoop.fs.Path;
import java.io.IOException;
-import java.util.Objects;
/**
* Base class for {@link Writer Writers} that write to a {@link FSDataOutputStream}.
@@ -102,24 +101,7 @@ public void close() throws IOException {
}
}
- @Override
- public int hashCode() {
- return Boolean.hashCode(syncOnFlush);
- }
-
- @Override
- public boolean equals(Object other) {
- if (this == other) {
- return true;
- }
- if (other == null) {
- return false;
- }
- if (getClass() != other.getClass()) {
- return false;
- }
- StreamWriterBase<T> writer = (StreamWriterBase<T>) other;
- // field comparison
- return Objects.equals(syncOnFlush, writer.syncOnFlush);
+ public boolean isSyncOnFlush() {
+ return syncOnFlush;
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StringWriter.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StringWriter.java
index 5c81b15bf1a..122bc7fc195 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StringWriter.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/StringWriter.java
@@ -26,7 +26,6 @@
import java.nio.charset.Charset;
import java.nio.charset.IllegalCharsetNameException;
import java.nio.charset.UnsupportedCharsetException;
-import java.util.Objects;
/**
* A {@link Writer} that uses {@code toString()} on the input elements and writes them to
@@ -87,29 +86,11 @@ public void write(T element) throws IOException {
}
@Override
- public Writer<T> duplicate() {
+ public StringWriter<T> duplicate() {
return new StringWriter<>(this);
}
- @Override
- public int hashCode() {
- return Objects.hash(super.hashCode(), charsetName);
- }
-
- @Override
- public boolean equals(Object other) {
- if (this == other) {
- return true;
- }
- if (other == null) {
- return false;
- }
- if (getClass() != other.getClass()) {
- return false;
- }
- StringWriter<T> writer = (StringWriter<T>) other;
- // field comparison
- return Objects.equals(charsetName, writer.charsetName)
- && super.equals(other);
+ String getCharsetName() {
+ return charsetName;
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
index 594ff873b70..17e5d56298b 100644
--- a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
+++ b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
@@ -88,9 +88,11 @@
* and a rolling counter. For example the file {@code "part-1-17"} contains the data from
* {@code subtask 1} of the sink and is the {@code 17th} bucket created by that subtask. Per default
* the part prefix is {@code "part"} but this can be configured using {@link #setPartPrefix(String)}.
- * When a part file becomes bigger than the user-specified batch size the current part file is closed,
- * the part counter is increased and a new part file is created. The batch size defaults to {@code 384MB},
- * this can be configured using {@link #setBatchSize(long)}.
+ * When a part file becomes bigger than the user-specified batch size or when the part file becomes older
+ * than the user-specified roll over interval the current part file is closed, the part counter is increased
+ * and a new part file is created. The batch size defaults to {@code 384MB}, this can be configured
+ * using {@link #setBatchSize(long)}. The roll over interval defaults to {@code Long.MAX_VALUE} and
+ * this can be configured using {@link #setBatchRolloverInterval(long)}.
*
*
* <p>In some scenarios, the open buckets are required to change based on time. In these cases, the sink
@@ -137,6 +139,10 @@
* {@link #setWriter(Writer)}. For example, {@link SequenceFileWriter}
* can be used to write Hadoop {@code SequenceFiles}.
* </li>
+ * <li>
+ * {@link #closePartFilesByTime(long)} closes buckets that have not been written to for
+ * {@code inactiveBucketThreshold} or if they are older than {@code batchRolloverInterval}.
+ * </li>
* </ol>
*
*
@@ -239,6 +245,11 @@
*/
private static final long DEFAULT_ASYNC_TIMEOUT_MS = 60 * 1000;
+ /**
+ * The default time interval at which part files are written to the filesystem.
+ */
+ private static final long DEFAULT_BATCH_ROLLOVER_INTERVAL = Long.MAX_VALUE;
+
/**
* The base {@code Path} that stores all bucket directories.
*/
@@ -258,6 +269,7 @@
private long batchSize = DEFAULT_BATCH_SIZE;
private long inactiveBucketCheckInterval = DEFAULT_INACTIVE_BUCKET_CHECK_INTERVAL_MS;
private long inactiveBucketThreshold = DEFAULT_INACTIVE_BUCKET_THRESHOLD_MS;
+ private long batchRolloverInterval = DEFAULT_BATCH_ROLLOVER_INTERVAL;
// These are the actually configured prefixes/suffixes
private String inProgressSuffix = DEFAULT_IN_PROGRESS_SUFFIX;
@@ -442,7 +454,7 @@ public void invoke(T value) throws Exception {
state.addBucketState(bucketPath, bucketState);
}
- if (shouldRoll(bucketState)) {
+ if (shouldRoll(bucketState, currentProcessingTime)) {
openNewPartFile(bucketPath, bucketState);
}
@@ -456,9 +468,10 @@ public void invoke(T value) throws Exception {
* <ol>
* <li>no file is created yet for the task to write to, or</li>
* <li>the current file has reached the maximum bucket size.</li>
+ * <li>the current file is older than roll over interval</li>
* </ol>
*/
- private boolean shouldRoll(BucketState<T> bucketState) throws IOException {
+ private boolean shouldRoll(BucketState<T> bucketState, long currentProcessingTime) throws IOException {
boolean shouldRoll = false;
int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
if (!bucketState.isWriterOpen) {
@@ -473,6 +486,14 @@ private boolean shouldRoll(BucketState<T> bucketState) throws IOException {
subtaskIndex,
writePosition,
batchSize);
+ } else {
+ if (currentProcessingTime - bucketState.creationTime > batchRolloverInterval) {
+ shouldRoll = true;
+ LOG.debug(
+ "BucketingSink {} starting new bucket because file is older than roll over interval {}.",
+ subtaskIndex,
+ batchRolloverInterval);
+ }
}
}
return shouldRoll;
@@ -482,21 +503,23 @@ private boolean shouldRoll(BucketState<T> bucketState) throws IOException {
public void onProcessingTime(long timestamp) throws Exception {
long currentProcessingTime = processingTimeService.getCurrentProcessingTime();
- checkForInactiveBuckets(currentProcessingTime);
+ closePartFilesByTime(currentProcessingTime);
processingTimeService.registerTimer(currentProcessingTime + inactiveBucketCheckInterval, this);
}
/**
* Checks for inactive buckets, and closes them. Buckets are considered inactive if they have not been
- * written to for a period greater than {@code inactiveBucketThreshold} ms. This enables in-progress
- * files to be moved to the pending state and be finalised on the next checkpoint.
+ * written to for a period greater than {@code inactiveBucketThreshold} ms. Buckets are also closed if they are
+ * older than {@code batchRolloverInterval} ms. This enables in-progress files to be moved to the pending state
+ * and be finalised on the next checkpoint.
*/
- private void checkForInactiveBuckets(long currentProcessingTime) throws Exception {
+ private void closePartFilesByTime(long currentProcessingTime) throws Exception {
synchronized (state.bucketStates) {
for (Map.Entry<String, BucketState<T>> entry : state.bucketStates.entrySet()) {
- if (entry.getValue().lastWrittenToTime < currentProcessingTime - inactiveBucketThreshold) {
+ if ((entry.getValue().lastWrittenToTime < currentProcessingTime - inactiveBucketThreshold)
+ || (entry.getValue().creationTime < currentProcessingTime - batchRolloverInterval)) {
LOG.debug("BucketingSink {} closing bucket due to inactivity of over {} ms.",
getRuntimeContext().getIndexOfThisSubtask(), inactiveBucketThreshold);
closeCurrentPartFile(entry.getValue());
@@ -529,17 +552,16 @@ private void openNewPartFile(Path bucketPath, BucketState<T> bucketState) throws
// clean the base directory in case of rescaling.
int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
- Path partPath = new Path(bucketPath, partPrefix + "-" + subtaskIndex + "-" + bucketState.partCounter);
+ Path partPath = assemblePartPath(bucketPath, subtaskIndex, bucketState.partCounter);
while (fs.exists(partPath) ||
fs.exists(getPendingPathFor(partPath)) ||
fs.exists(getInProgressPathFor(partPath))) {
bucketState.partCounter++;
- partPath = new Path(bucketPath, partPrefix + "-" + subtaskIndex + "-" + bucketState.partCounter);
+ partPath = assemblePartPath(bucketPath, subtaskIndex, bucketState.partCounter);
}
- if (partSuffix != null) {
- partPath = partPath.suffix(partSuffix);
- }
+ // Record the creation time of the bucket
+ bucketState.creationTime = processingTimeService.getCurrentProcessingTime();
// increase, so we don't have to check for this name next time
bucketState.partCounter++;
@@ -550,6 +572,12 @@ private void openNewPartFile(Path bucketPath, BucketState<T> bucketState) throws
Path inProgressPath = getInProgressPathFor(partPath);
if (bucketState.writer == null) {
bucketState.writer = writerTemplate.duplicate();
+ if (bucketState.writer == null) {
+ throw new UnsupportedOperationException(
+ "Could not duplicate writer. " +
+ "Class '" + writerTemplate.getClass().getCanonicalName() + "' must implement the 'Writer.duplicate()' method."
+ );
+ }
}
bucketState.writer.open(fs, inProgressPath);
@@ -603,12 +631,9 @@ private Method reflectTruncate(FileSystem fs) {
}
// verify that truncate actually works
- FSDataOutputStream outputStream;
- Path testPath = new Path(UUID.randomUUID().toString());
- try {
- outputStream = fs.create(testPath);
+ Path testPath = new Path(basePath, UUID.randomUUID().toString());
+ try (FSDataOutputStream outputStream = fs.create(testPath)) {
outputStream.writeUTF("hello");
- outputStream.close();
} catch (IOException e) {
LOG.error("Could not create file for checking if truncate works.", e);
throw new RuntimeException("Could not create file for checking if truncate works. " +
@@ -635,6 +660,11 @@ private Method reflectTruncate(FileSystem fs) {
return m;
}
+ private Path assemblePartPath(Path bucket, int subtaskIndex, int partIndex) {
+ String localPartSuffix = partSuffix != null ? partSuffix : "";
+ return new Path(bucket, String.format("%s-%s-%s%s", partPrefix, subtaskIndex, partIndex, localPartSuffix));
+ }
+
private Path getPendingPathFor(Path path) {
return new Path(path.getParent(), pendingPrefix + path.getName()).suffix(pendingSuffix);
}
@@ -847,9 +877,9 @@ private void handlePendingInProgressFile(String file, long validLength) {
Path validLengthFilePath = getValidLengthPathFor(partPath);
if (!fs.exists(validLengthFilePath) && fs.exists(partPath)) {
LOG.debug("Writing valid-length file for {} to specify valid length {}", partPath, validLength);
- FSDataOutputStream lengthFileOut = fs.create(validLengthFilePath);
- lengthFileOut.writeUTF(Long.toString(validLength));
- lengthFileOut.close();
+ try (FSDataOutputStream lengthFileOut = fs.create(validLengthFilePath)) {
+ lengthFileOut.writeUTF(Long.toString(validLength));
+ }
}
}
@@ -908,6 +938,25 @@ private void handlePendingFilesForPreviousCheckpoints(Map<Long, List<String>> pe
return this;
}
+ /**
+ * Sets the roll over interval in milliseconds.
+ *
+ *
+ * <p>When a bucket part file is older than the roll over interval, a new bucket part file is
+ * started and the old one is closed. The name of the bucket file depends on the {@link Bucketer}.
+ * Additionally, the old part file is also closed if the bucket is not written to for a minimum of
+ * {@code inactiveBucketThreshold} ms.
+ *
+ * @param batchRolloverInterval The roll over interval in milliseconds
+ */
+ public BucketingSink<T> setBatchRolloverInterval(long batchRolloverInterval) {
+ if (batchRolloverInterval > 0) {
+ this.batchRolloverInterval = batchRolloverInterval;
+ }
+
+ return this;
+ }
+
/**
* Sets the default time between checks for inactive buckets.
*
@@ -921,6 +970,8 @@ private void handlePendingFilesForPreviousCheckpoints(Map<Long, List<String>> pe
/**
* Sets the default threshold for marking a bucket as inactive and closing its part files.
* Buckets which haven't been written to for at least this period of time become inactive.
+ * Additionally, part files for the bucket are also closed if the bucket is older than
+ * {@code batchRolloverInterval} ms.
*
* @param threshold The timeout, in milliseconds.
*/
@@ -1110,6 +1161,11 @@ public String toString() {
*/
long lastWrittenToTime;
+ /**
+ * The time this bucket was created.
+ */
+ long creationTime;
+
/**
* Pending files that accumulated since the last checkpoint.
*/
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriterTest.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriterTest.java
index 019e56d66ba..864d9c1b6fa 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriterTest.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/AvroKeyValueSinkWriterTest.java
@@ -18,8 +18,6 @@
package org.apache.flink.streaming.connectors.fs;
-import org.apache.flink.api.java.tuple.Tuple2;
-
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileConstants;
import org.junit.Test;
@@ -47,11 +45,11 @@ public void testDuplicate() {
AvroKeyValueSinkWriter<String, String> writer = new AvroKeyValueSinkWriter(properties);
writer.setSyncOnFlush(true);
- Writer<Tuple2<String, String>> other = writer.duplicate();
+ AvroKeyValueSinkWriter<String, String> other = writer.duplicate();
- assertTrue(writer.equals(other));
+ assertTrue(StreamWriterBaseComparator.equals(writer, other));
writer.setSyncOnFlush(false);
- assertFalse(writer.equals(other));
+ assertFalse(StreamWriterBaseComparator.equals(writer, other));
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkITCase.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkITCase.java
index 78b53d8de52..86821a5dd2f 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkITCase.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkITCase.java
@@ -33,6 +33,7 @@
import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
import org.apache.flink.test.util.MiniClusterResource;
+import org.apache.flink.test.util.MiniClusterResourceConfiguration;
import org.apache.flink.util.Collector;
import org.apache.flink.util.NetUtils;
import org.apache.flink.util.TestLogger;
@@ -118,10 +119,10 @@ public static void setup() throws Exception {
+ "/";
miniClusterResource = new MiniClusterResource(
- new MiniClusterResource.MiniClusterResourceConfiguration(
- new org.apache.flink.configuration.Configuration(),
- 1,
- 4));
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(4)
+ .build());
miniClusterResource.before();
}
@@ -929,7 +930,7 @@ public void open(FileSystem fs, Path path) throws IOException {
}
@Override
- public Writer<T> duplicate() {
+ public StreamWriterWithConfigCheck<T> duplicate() {
return new StreamWriterWithConfigCheck<>(key, expect);
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
index b9564ee32d5..72962696945 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
@@ -26,6 +26,7 @@
import org.apache.flink.runtime.security.SecurityConfiguration;
import org.apache.flink.runtime.security.modules.HadoopModule;
import org.apache.flink.test.util.MiniClusterResource;
+import org.apache.flink.test.util.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.SecureTestEnvironment;
import org.apache.flink.test.util.TestBaseUtils;
import org.apache.flink.test.util.TestingSecurityContext;
@@ -147,10 +148,12 @@ public static void setup() throws Exception {
Configuration configuration = startSecureFlinkClusterWithRecoveryModeEnabled();
- miniClusterResource = new MiniClusterResource(new MiniClusterResource.MiniClusterResourceConfiguration(
- configuration,
- 1,
- 4));
+ miniClusterResource = new MiniClusterResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setConfiguration(configuration)
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(4)
+ .build());
miniClusterResource.before();
}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriterTest.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriterTest.java
index 7ea22649d49..44716d3b4fe 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriterTest.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/SequenceFileWriterTest.java
@@ -18,8 +18,6 @@
package org.apache.flink.streaming.connectors.fs;
-import org.apache.flink.api.java.tuple.Tuple2;
-
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.junit.Test;
@@ -36,11 +34,11 @@
public void testDuplicate() {
SequenceFileWriter<Text, Text> writer = new SequenceFileWriter("BZ", SequenceFile.CompressionType.BLOCK);
writer.setSyncOnFlush(true);
- Writer<Tuple2<Text, Text>> other = writer.duplicate();
+ SequenceFileWriter<Text, Text> other = writer.duplicate();
- assertTrue(writer.equals(other));
+ assertTrue(StreamWriterBaseComparator.equals(writer, other));
writer.setSyncOnFlush(false);
- assertFalse(writer.equals(other));
+ assertFalse(StreamWriterBaseComparator.equals(writer, other));
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StreamWriterBaseComparator.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StreamWriterBaseComparator.java
new file mode 100644
index 00000000000..9472c29ee45
--- /dev/null
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StreamWriterBaseComparator.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.fs;
+
+import org.apache.hadoop.io.Writable;
+
+import java.util.Objects;
+
+/**
+ * Helper class to perform partial comparisons of {@link StreamWriterBase} instances. During comparisons
+ * it ignores changes in underlying output streams.
+ */
+public class StreamWriterBaseComparator {
+
+ public static <T> boolean equals(
+ StreamWriterBase<T> writer1,
+ StreamWriterBase<T> writer2) {
+ return Objects.equals(writer1.isSyncOnFlush(), writer2.isSyncOnFlush());
+ }
+
+ public static <K, V> boolean equals(
+ AvroKeyValueSinkWriter<K, V> writer1,
+ AvroKeyValueSinkWriter<K, V> writer2) {
+ return equals((StreamWriterBase) writer1, (StreamWriterBase) writer2) &&
+ Objects.equals(writer1.getProperties(), writer2.getProperties());
+ }
+
+ public static <K extends Writable, V extends Writable> boolean equals(
+ SequenceFileWriter<K, V> writer1,
+ SequenceFileWriter<K, V> writer2) {
+ return equals((StreamWriterBase) writer1, (StreamWriterBase) writer2) &&
+ Objects.equals(writer1.getCompressionCodecName(), writer2.getCompressionCodecName()) &&
+ Objects.equals(writer1.getCompressionType(), writer2.getCompressionType()) &&
+ Objects.equals(writer1.getKeyClass(), writer2.getKeyClass()) &&
+ Objects.equals(writer1.getValueClass(), writer2.getValueClass());
+ }
+
+ public static <T> boolean equals(
+ StringWriter<T> writer1,
+ StringWriter<T> writer2) {
+ return equals((StreamWriterBase) writer1, (StreamWriterBase) writer2) &&
+ Objects.equals(writer1.getCharsetName(), writer2.getCharsetName());
+ }
+}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StringWriterTest.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StringWriterTest.java
index 488f860c327..7009d9467fb 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StringWriterTest.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/StringWriterTest.java
@@ -34,12 +34,12 @@
public void testDuplicate() {
StringWriter<String> writer = new StringWriter(StandardCharsets.UTF_16.name());
writer.setSyncOnFlush(true);
- Writer<String> other = writer.duplicate();
+ StringWriter<String> other = writer.duplicate();
- assertTrue(writer.equals(other));
+ assertTrue(StreamWriterBaseComparator.equals(writer, other));
writer.setSyncOnFlush(false);
- assertFalse(writer.equals(other));
- assertFalse(writer.equals(new StringWriter<>()));
+ assertFalse(StreamWriterBaseComparator.equals(writer, other));
+ assertFalse(StreamWriterBaseComparator.equals(writer, new StringWriter<>()));
}
}
diff --git a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSinkTest.java b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSinkTest.java
index 56415feab0e..ee425ff811e 100644
--- a/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSinkTest.java
+++ b/flink-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSinkTest.java
@@ -28,7 +28,6 @@
import org.apache.flink.streaming.connectors.fs.Clock;
import org.apache.flink.streaming.connectors.fs.SequenceFileWriter;
import org.apache.flink.streaming.connectors.fs.StringWriter;
-import org.apache.flink.streaming.connectors.fs.Writer;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
@@ -65,6 +64,8 @@
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
+import java.nio.file.Files;
+import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;
@@ -73,6 +74,8 @@
import static org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkTestUtils.PENDING_SUFFIX;
import static org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkTestUtils.VALID_LENGTH_SUFFIX;
import static org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkTestUtils.checkLocalFs;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
/**
* Tests for the {@link BucketingSink}.
@@ -137,6 +140,33 @@ public Path getBucketPath(Clock clock, Path basePath, String element) {
return new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 10, totalParallelism, taskIdx);
}
+ private OneInputStreamOperatorTestHarness<String, Object> createRescalingTestSinkWithRollover(
+ File outDir, int totalParallelism, int taskIdx, long inactivityInterval, long rolloverInterval) throws Exception {
+
+ BucketingSink<String> sink = new BucketingSink<String>(outDir.getAbsolutePath())
+ .setBucketer(new Bucketer<String>() {
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public Path getBucketPath(Clock clock, Path basePath, String element) {
+ return new Path(basePath, element);
+ }
+ })
+ .setWriter(new StringWriter<String>())
+ .setInactiveBucketCheckInterval(inactivityInterval)
+ .setInactiveBucketThreshold(inactivityInterval)
+ .setPartPrefix(PART_PREFIX)
+ .setInProgressPrefix("")
+ .setPendingPrefix("")
+ .setValidLengthPrefix("")
+ .setInProgressSuffix(IN_PROGRESS_SUFFIX)
+ .setPendingSuffix(PENDING_SUFFIX)
+ .setValidLengthSuffix(VALID_LENGTH_SUFFIX)
+ .setBatchRolloverInterval(rolloverInterval);
+
+ return createTestSink(sink, totalParallelism, taskIdx);
+ }
+
@BeforeClass
public static void createHDFS() throws IOException {
Assume.assumeTrue("HDFS cluster cannot be started on Windows without extensions.", !OperatingSystem.isWindows());
@@ -437,6 +467,40 @@ public void testScalingUp() throws Exception {
checkLocalFs(outDir, 0, 3, 5, 5);
}
+ @Test
+ public void testRolloverInterval() throws Exception {
+ final File outDir = tempFolder.newFolder();
+
+ OneInputStreamOperatorTestHarness<String, Object> testHarness = createRescalingTestSinkWithRollover(outDir, 1, 0, 1000L, 100L);
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.setProcessingTime(0L);
+
+ testHarness.processElement(new StreamRecord<>("test1", 1L));
+ checkLocalFs(outDir, 1, 0, 0, 0);
+
+ // invoke rollover based on rollover interval
+ testHarness.setProcessingTime(101L);
+ testHarness.processElement(new StreamRecord<>("test1", 2L));
+ checkLocalFs(outDir, 1, 1, 0, 0);
+
+ testHarness.snapshot(0, 0);
+ testHarness.notifyOfCompletedCheckpoint(0);
+ checkLocalFs(outDir, 1, 0, 1, 0);
+
+ // move the in-progress file to pending
+ testHarness.setProcessingTime(3000L);
+ testHarness.snapshot(1, 1);
+ checkLocalFs(outDir, 0, 1, 1, 0);
+
+ // move the pending file to "committed"
+ testHarness.notifyOfCompletedCheckpoint(1);
+ testHarness.close();
+
+ checkLocalFs(outDir, 0, 0, 2, 0);
+ }
+
/**
* This tests {@link StringWriter} with
* non-bucketing output.
@@ -839,6 +903,81 @@ public void testUserDefinedConfiguration() throws Exception {
inStream.close();
}
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsSpecifiedAndPreviousPartFileInProgressState()
+ throws Exception {
+ testThatPartIndexIsIncremented(".my", "part-0-0.my" + IN_PROGRESS_SUFFIX);
+ }
+
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsSpecifiedAndPreviousPartFileInPendingState()
+ throws Exception {
+ testThatPartIndexIsIncremented(".my", "part-0-0.my" + PENDING_SUFFIX);
+ }
+
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsSpecifiedAndPreviousPartFileInFinalState()
+ throws Exception {
+ testThatPartIndexIsIncremented(".my", "part-0-0.my");
+ }
+
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsNotSpecifiedAndPreviousPartFileInProgressState()
+ throws Exception {
+ testThatPartIndexIsIncremented(null, "part-0-0" + IN_PROGRESS_SUFFIX);
+ }
+
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsNotSpecifiedAndPreviousPartFileInPendingState()
+ throws Exception {
+ testThatPartIndexIsIncremented(null, "part-0-0" + PENDING_SUFFIX);
+ }
+
+ @Test
+ public void testThatPartIndexIsIncrementedWhenPartSuffixIsNotSpecifiedAndPreviousPartFileInFinalState()
+ throws Exception {
+ testThatPartIndexIsIncremented(null, "part-0-0");
+ }
+
+ private void testThatPartIndexIsIncremented(String partSuffix, String existingPartFile) throws Exception {
+ File outDir = tempFolder.newFolder();
+ long inactivityInterval = 100;
+
+ java.nio.file.Path bucket = Paths.get(outDir.getPath());
+ Files.createFile(bucket.resolve(existingPartFile));
+
+ String basePath = outDir.getAbsolutePath();
+ BucketingSink<String> sink = new BucketingSink<String>(basePath)
+ .setBucketer(new BasePathBucketer<>())
+ .setInactiveBucketCheckInterval(inactivityInterval)
+ .setInactiveBucketThreshold(inactivityInterval)
+ .setPartPrefix(PART_PREFIX)
+ .setInProgressPrefix("")
+ .setPendingPrefix("")
+ .setValidLengthPrefix("")
+ .setInProgressSuffix(IN_PROGRESS_SUFFIX)
+ .setPendingSuffix(PENDING_SUFFIX)
+ .setValidLengthSuffix(VALID_LENGTH_SUFFIX)
+ .setPartSuffix(partSuffix)
+ .setBatchSize(0);
+
+ try (OneInputStreamOperatorTestHarness<String, Object> testHarness = createTestSink(sink, 1, 0)) {
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.setProcessingTime(0L);
+
+ testHarness.processElement(new StreamRecord<>("test1", 1L));
+
+ testHarness.setProcessingTime(101L);
+ testHarness.snapshot(0, 0);
+ testHarness.notifyOfCompletedCheckpoint(0);
+ }
+
+ String expectedFileName = partSuffix == null ? "part-0-1" : "part-0-1" + partSuffix;
+ assertThat(Files.exists(bucket.resolve(expectedFileName)), is(true));
+ }
+
private static class StreamWriterWithConfigCheck<K, V> extends AvroKeyValueSinkWriter<K, V> {
private Map<String, String> properties;
private String key;
@@ -857,7 +996,7 @@ public void open(FileSystem fs, Path path) throws IOException {
}
@Override
- public Writer<Tuple2<K, V>> duplicate() {
+ public StreamWriterWithConfigCheck<K, V> duplicate() {
return new StreamWriterWithConfigCheck<>(properties, key, expect);
}
}
diff --git a/flink-connectors/flink-connector-kafka-0.10/pom.xml b/flink-connectors/flink-connector-kafka-0.10/pom.xml
index 22efc3470d7..135dc59b655 100644
--- a/flink-connectors/flink-connector-kafka-0.10/pom.xml
+++ b/flink-connectors/flink-connector-kafka-0.10/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
@@ -202,6 +202,59 @@ under the License.
</dependencies>
+ <profiles>
+ <!-- Create SQL Client uber jars by default -->
+ <profile>
+ <id>sql-jars</id>
+ <activation>
+ <property>
+ <name>!skipSqlJars</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadedArtifactAttached>true</shadedArtifactAttached>
+ <shadedClassifierName>sql-jar</shadedClassifierName>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.kafka:*</include>
+ <include>org.apache.flink:flink-connector-kafka-base_${scala.binary.version}</include>
+ <include>org.apache.flink:flink-connector-kafka-0.9_${scala.binary.version}</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>kafka/kafka-version.properties</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+ <pattern>org.apache.kafka</pattern>
+ <shadedPattern>org.apache.flink.kafka010.shaded.org.apache.kafka</shadedPattern>
+ </relocation>
+ </relocations>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
<build>
<plugins>
<plugin>
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSource.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSource.java
index f759f61f8fb..d9d0a91ced1 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -35,8 +35,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.10.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka010AvroTableSource extends KafkaAvroTableSource {
/**
@@ -46,7 +51,9 @@
* @param properties Properties for the Kafka consumer.
* @param schema Schema of the produced table.
* @param record Avro specific record.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka010AvroTableSource(
String topic,
Properties properties,
@@ -69,7 +76,9 @@ public Kafka010AvroTableSource(
* the value to the field of the Avro record.</p>
*
* @param fieldMapping A mapping from schema fields to Avro fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -79,7 +88,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -89,7 +100,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -102,15 +115,27 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka010AvroTableSource}.
+ *
* @return A builder to configure and create a {@link Kafka010AvroTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Builder builder() {
return new Builder();
}
/**
* A builder to configure and create a {@link Kafka010AvroTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaAvroTableSource.Builder<Kafka010AvroTableSource, Kafka010AvroTableSource.Builder> {
@Override
@@ -127,7 +152,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka010AvroTableSource}.
*
* @return A configured {@link Kafka010AvroTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka010AvroTableSource build() {
Kafka010AvroTableSource tableSource = new Kafka010AvroTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSink.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSink.java
index ef33cd56237..2ad31420789 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSink.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSink.java
@@ -18,18 +18,23 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.types.Row;
import java.util.Properties;
/**
* Kafka 0.10 {@link KafkaTableSink} that serializes data in JSON format.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka010JsonTableSink extends KafkaJsonTableSink {
/**
@@ -46,7 +51,9 @@
*
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka010JsonTableSink(String topic, Properties properties) {
super(topic, properties, new FlinkFixedPartitioner<>());
}
@@ -58,14 +65,20 @@ public Kafka010JsonTableSink(String topic, Properties properties) {
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
* @param partitioner Kafka partitioner
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka010JsonTableSink(String topic, Properties properties, FlinkKafkaPartitioner<Row> partitioner) {
super(topic, properties, partitioner);
}
@Override
protected FlinkKafkaProducerBase<Row> createKafkaProducer(String topic, Properties properties, SerializationSchema<Row> serializationSchema, FlinkKafkaPartitioner<Row> partitioner) {
- return new FlinkKafkaProducer010<>(topic, serializationSchema, properties, partitioner);
+ return new FlinkKafkaProducer010<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSource.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSource.java
index bda236f7445..38d9034a9a7 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -32,8 +32,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.10.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka010JsonTableSource extends KafkaJsonTableSource {
/**
@@ -43,7 +48,9 @@
* @param properties Properties for the Kafka consumer.
* @param tableSchema The schema of the table.
* @param jsonSchema The schema of the JSON messages to decode from Kafka.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka010JsonTableSource(
String topic,
Properties properties,
@@ -58,7 +65,9 @@ public Kafka010JsonTableSource(
* TableSource will fail for missing fields if set to true. If set to false, the missing field is set to null.
*
* @param failOnMissingField Flag that specifies the TableSource behavior in case of missing fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFailOnMissingField(boolean failOnMissingField) {
super.setFailOnMissingField(failOnMissingField);
@@ -68,7 +77,9 @@ public void setFailOnMissingField(boolean failOnMissingField) {
* Sets the mapping from table schema fields to JSON schema fields.
*
* @param fieldMapping The mapping from table schema fields to JSON schema fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -78,7 +89,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -88,7 +101,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -101,15 +116,27 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka010JsonTableSource}.
+ *
* @return A builder to configure and create a {@link Kafka010JsonTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Kafka010JsonTableSource.Builder builder() {
return new Kafka010JsonTableSource.Builder();
}
/**
* A builder to configure and create a {@link Kafka010JsonTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaJsonTableSource.Builder<Kafka010JsonTableSource, Kafka010JsonTableSource.Builder> {
@Override
@@ -126,6 +153,7 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka010JsonTableSource}.
*
* @return A configured {@link Kafka010JsonTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
@Override
public Kafka010JsonTableSource build() {
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSink.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSink.java
new file mode 100644
index 00000000000..a8c65539824
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSink.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+
+import java.util.Properties;
+
+/**
+ * Kafka 0.10 table sink for writing data into Kafka.
+ */
+@Internal
+public class Kafka010TableSink extends KafkaTableSink {
+
+ public Kafka010TableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+ super(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+
+ @Override
+ protected FlinkKafkaProducerBase<Row> createKafkaProducer(
+ String topic,
+ Properties properties,
+ SerializationSchema<Row> serializationSchema,
+ FlinkKafkaPartitioner<Row> partitioner) {
+ return new FlinkKafkaProducer010<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSource.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSource.java
index 5a0222794fd..f6574623b2a 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSource.java
@@ -18,48 +18,79 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
import java.util.Properties;
/**
* Kafka {@link StreamTableSource} for Kafka 0.10.
*/
-@PublicEvolving
-public abstract class Kafka010TableSource extends KafkaTableSource {
-
- // The deserialization schema for the Kafka records
- private final DeserializationSchema<Row> deserializationSchema;
+@Internal
+public class Kafka010TableSource extends KafkaTableSource {
/**
* Creates a Kafka 0.10 {@link StreamTableSource}.
*
- * @param topic Kafka topic to consume.
- * @param properties Properties for the Kafka consumer.
- * @param deserializationSchema Deserialization schema to use for Kafka records.
- * @param typeInfo Type information describing the result type. The field names are used
- * to parse the JSON file and so are the types.
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*/
public Kafka010TableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Optional<Map<String, String>> fieldMapping,
String topic,
Properties properties,
DeserializationSchema<Row> deserializationSchema,
- TableSchema schema,
- TypeInformation<Row> typeInfo) {
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
- super(topic, properties, schema, typeInfo);
-
- this.deserializationSchema = deserializationSchema;
+ super(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ fieldMapping,
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
}
- @Override
- public DeserializationSchema<Row> getDeserializationSchema() {
- return this.deserializationSchema;
+ /**
+ * Creates a Kafka 0.10 {@link StreamTableSource}.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ */
+ public Kafka010TableSource(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema) {
+
+ super(schema, topic, properties, deserializationSchema);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactory.java b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactory.java
new file mode 100644
index 00000000000..0cf94995bdd
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Factory for creating configured instances of {@link Kafka010TableSource}.
+ */
+public class Kafka010TableSourceSinkFactory extends KafkaTableSourceSinkFactoryBase {
+
+ @Override
+ protected String kafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_010;
+ }
+
+ @Override
+ protected boolean supportsKafkaTimestamps() {
+ return true;
+ }
+
+ @Override
+ protected KafkaTableSource createKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka010TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
+ }
+
+ @Override
+ protected KafkaTableSink createKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka010TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory b/flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
similarity index 91%
rename from flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
rename to flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
index 9ef54fcb045..9bb036302f8 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
+++ b/flink-connectors/flink-connector-kafka-0.10/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.flink.streaming.connectors.kafka.Kafka010JsonTableSourceFactory
+org.apache.flink.streaming.connectors.kafka.Kafka010TableSourceSinkFactory
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSourceTest.java
index f5f8af138d8..bf253c43ece 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010AvroTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka010AvroTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka010AvroTableSourceTest extends KafkaAvroTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java
index a038c8e7291..45ab95842e2 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java
@@ -21,9 +21,9 @@
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.serialization.TypeInformationSerializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.java.typeutils.GenericTypeInfo;
-import org.apache.flink.api.java.typeutils.TypeInfoParser;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
import org.apache.flink.streaming.api.TimeCharacteristic;
@@ -39,6 +39,7 @@
import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema;
import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
+import org.junit.Ignore;
import org.junit.Test;
import javax.annotation.Nullable;
@@ -169,6 +170,7 @@ public void testAutoOffsetRetrievalAndCommitToKafka() throws Exception {
/**
* Kafka 0.10 specific test, ensuring Timestamps are properly written to and read from Kafka.
*/
+ @Ignore("This test is disabled because of: https://issues.apache.org/jira/browse/FLINK-9217")
@Test(timeout = 60000)
public void testTimestamps() throws Exception {
@@ -204,7 +206,7 @@ public void cancel() {
}
});
- final TypeInformationSerializationSchema<Long> longSer = new TypeInformationSerializationSchema<>(TypeInfoParser.<Long>parse("Long"), env.getConfig());
+ final TypeInformationSerializationSchema<Long> longSer = new TypeInformationSerializationSchema<>(Types.LONG, env.getConfig());
FlinkKafkaProducer010.FlinkKafkaProducer010Configuration prod = FlinkKafkaProducer010.writeToKafkaWithTimestamps(streamWithTimestamps, topic, new KeyedSerializationSchemaWrapper<>(longSer), standardProps, new FlinkKafkaPartitioner<Long>() {
private static final long serialVersionUID = -6730989584364230617L;
@@ -316,7 +318,7 @@ public void close() throws Exception {
long cnt = 0;
public LimitedLongDeserializer() {
- this.ti = TypeInfoParser.parse("Long");
+ this.ti = Types.LONG;
this.ser = ti.createSerializer(new ExecutionConfig());
}
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSinkTest.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSinkTest.java
index af562c6fa9e..339420cede3 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSinkTest.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSinkTest.java
@@ -27,7 +27,11 @@
/**
* Tests for the {@link Kafka010JsonTableSink}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sinks.
*/
+@Deprecated
public class Kafka010JsonTableSinkTest extends KafkaTableSinkTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceFactoryTest.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceFactoryTest.java
index 22cf659ffeb..ab83a01c48f 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceFactoryTest.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceFactoryTest.java
@@ -21,8 +21,12 @@
import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_VERSION_VALUE_010;
/**
- * Tests for {@link Kafka010JsonTableSourceFactory}.
+ * Tests for legacy Kafka010JsonTableSourceFactory.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka010JsonTableSourceFactoryTest extends KafkaJsonTableSourceFactoryTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceTest.java
index f9d4d84b3bf..087f3edc513 100644
--- a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010JsonTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka010JsonTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka010JsonTableSourceTest extends KafkaJsonTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactoryTest.java b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactoryTest.java
new file mode 100644
index 00000000000..cc198c91595
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010TableSourceSinkFactoryTest.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Test for {@link Kafka010TableSource} and {@link Kafka010TableSink} created
+ * by {@link Kafka010TableSourceSinkFactory}.
+ */
+public class Kafka010TableSourceSinkFactoryTest extends KafkaTableSourceSinkFactoryTestBase {
+
+ @Override
+ protected String getKafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_010;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ protected Class<FlinkKafkaConsumerBase<Row>> getExpectedFlinkKafkaConsumer() {
+ return (Class) FlinkKafkaConsumer010.class;
+ }
+
+ @Override
+ protected Class<?> getExpectedFlinkKafkaProducer() {
+ return FlinkKafkaProducer010.class;
+ }
+
+ @Override
+ protected KafkaTableSource getExpectedKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka010TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets
+ );
+ }
+
+ @Override
+ protected KafkaTableSink getExpectedKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka010TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema
+ );
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.11/pom.xml b/flink-connectors/flink-connector-kafka-0.11/pom.xml
index 8b404390235..bf04aebb390 100644
--- a/flink-connectors/flink-connector-kafka-0.11/pom.xml
+++ b/flink-connectors/flink-connector-kafka-0.11/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
@@ -212,9 +212,14 @@ under the License.
</dependencies>
<profiles>
+ <!-- Create SQL Client uber jars by default -->
<profile>
- <!-- Create SQL Client uber jars for releases -->
- <id>release</id>
+ <id>sql-jars</id>
+ <activation>
+ <property>
+ <name>!skipSqlJars</name>
+ </property>
+ </activation>
<build>
<plugins>
<plugin>
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java
index e92f38b3ea0..84973726113 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java
@@ -651,6 +651,17 @@ public void close() throws FlinkKafka011Exception {
if (currentTransaction != null) {
// to avoid exceptions on aborting transactions with some pending records
flush(currentTransaction);
+
+ // normal abort for AT_LEAST_ONCE and NONE do not clean up resources because of producer reusing, thus
+ // we need to close it manually
+ switch (semantic) {
+ case EXACTLY_ONCE:
+ break;
+ case AT_LEAST_ONCE:
+ case NONE:
+ currentTransaction.producer.close();
+ break;
+ }
}
try {
super.close();
@@ -826,7 +837,7 @@ public void initializeState(FunctionInitializationContext context) throws Except
nextTransactionalIdHintState = context.getOperatorStateStore().getUnionListState(
NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR);
transactionalIdsGenerator = new TransactionalIdsGenerator(
- getRuntimeContext().getTaskName(),
+ getRuntimeContext().getTaskName() + "-" + ((StreamingRuntimeContext) getRuntimeContext()).getOperatorUniqueID(),
getRuntimeContext().getIndexOfThisSubtask(),
getRuntimeContext().getNumberOfParallelSubtasks(),
kafkaProducersPoolSize,
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSource.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSource.java
index f4484f6fabb..fab592f1bda 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -35,8 +35,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.11.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka011AvroTableSource extends KafkaAvroTableSource {
/**
@@ -46,7 +51,9 @@
* @param properties Properties for the Kafka consumer.
* @param schema Schema of the produced table.
* @param record Avro specific record.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka011AvroTableSource(
String topic,
Properties properties,
@@ -69,7 +76,9 @@ public Kafka011AvroTableSource(
* the value to the field of the Avro record.</p>
*
* @param fieldMapping A mapping from schema fields to Avro fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -79,7 +88,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -89,7 +100,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -103,14 +116,25 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka011AvroTableSource}.
* @return A builder to configure and create a {@link Kafka011AvroTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Builder builder() {
return new Builder();
}
/**
* A builder to configure and create a {@link Kafka011AvroTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaAvroTableSource.Builder<Kafka011AvroTableSource, Kafka011AvroTableSource.Builder> {
@Override
@@ -127,7 +151,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka011AvroTableSource}.
*
* @return A configured {@link Kafka011AvroTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka011AvroTableSource build() {
Kafka011AvroTableSource tableSource = new Kafka011AvroTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSource.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSource.java
index a012f5d4edd..375eeadffb9 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -32,8 +32,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.11.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka011JsonTableSource extends KafkaJsonTableSource {
/**
@@ -43,7 +48,9 @@
* @param properties Properties for the Kafka consumer.
* @param tableSchema The schema of the table.
* @param jsonSchema The schema of the JSON messages to decode from Kafka.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka011JsonTableSource(
String topic,
Properties properties,
@@ -58,7 +65,9 @@ public Kafka011JsonTableSource(
* TableSource will fail for missing fields if set to true. If set to false, the missing field is set to null.
*
* @param failOnMissingField Flag that specifies the TableSource behavior in case of missing fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFailOnMissingField(boolean failOnMissingField) {
super.setFailOnMissingField(failOnMissingField);
@@ -68,7 +77,9 @@ public void setFailOnMissingField(boolean failOnMissingField) {
* Sets the mapping from table schema fields to JSON schema fields.
*
* @param fieldMapping The mapping from table schema fields to JSON schema fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -78,7 +89,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -88,7 +101,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -101,15 +116,27 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka011JsonTableSource}.
+ *
* @return A builder to configure and create a {@link Kafka011JsonTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Kafka011JsonTableSource.Builder builder() {
return new Kafka011JsonTableSource.Builder();
}
/**
* A builder to configure and create a {@link Kafka011JsonTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaJsonTableSource.Builder<Kafka011JsonTableSource, Kafka011JsonTableSource.Builder> {
@Override
@@ -126,7 +153,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka011JsonTableSource}.
*
* @return A configured {@link Kafka011JsonTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka011JsonTableSource build() {
Kafka011JsonTableSource tableSource = new Kafka011JsonTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSink.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSink.java
new file mode 100644
index 00000000000..22c6da13b05
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSink.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Kafka 0.11 table sink for writing data into Kafka.
+ */
+@Internal
+public class Kafka011TableSink extends KafkaTableSink {
+
+ public Kafka011TableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+ super(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+
+ @Override
+ protected SinkFunction<Row> createKafkaProducer(
+ String topic,
+ Properties properties,
+ SerializationSchema<Row> serializationSchema,
+ FlinkKafkaPartitioner<Row> partitioner) {
+ return new FlinkKafkaProducer011<>(
+ topic,
+ new KeyedSerializationSchemaWrapper<>(serializationSchema),
+ properties,
+ Optional.of(partitioner));
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSource.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSource.java
index 6c9c37d79e3..a646317fe0a 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSource.java
@@ -18,53 +18,83 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
import java.util.Properties;
/**
* Kafka {@link StreamTableSource} for Kafka 0.11.
*/
-@PublicEvolving
-public abstract class Kafka011TableSource extends KafkaTableSource {
-
- // The deserialization schema for the Kafka records
- private final DeserializationSchema<Row> deserializationSchema;
+@Internal
+public class Kafka011TableSource extends KafkaTableSource {
/**
* Creates a Kafka 0.11 {@link StreamTableSource}.
*
- * @param topic Kafka topic to consume.
- * @param properties Properties for the Kafka consumer.
- * @param deserializationSchema Deserialization schema to use for Kafka records.
- * @param typeInfo Type information describing the result type. The field names are used
- * to parse the JSON file and so are the types.
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*/
public Kafka011TableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Optional<Map<String, String>> fieldMapping,
String topic,
Properties properties,
DeserializationSchema<Row> deserializationSchema,
- TableSchema schema,
- TypeInformation<Row> typeInfo) {
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
- super(topic, properties, schema, typeInfo);
-
- this.deserializationSchema = deserializationSchema;
+ super(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ fieldMapping,
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
}
- @Override
- public DeserializationSchema<Row> getDeserializationSchema() {
- return this.deserializationSchema;
+ /**
+ * Creates a Kafka 0.11 {@link StreamTableSource}.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ */
+ public Kafka011TableSource(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema) {
+
+ super(schema, topic, properties, deserializationSchema);
}
@Override
protected FlinkKafkaConsumerBase<Row> createKafkaConsumer(String topic, Properties properties, DeserializationSchema<Row> deserializationSchema) {
return new FlinkKafkaConsumer011<>(topic, deserializationSchema, properties);
}
-
}
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactory.java b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactory.java
new file mode 100644
index 00000000000..c26df42ed4a
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Factory for creating configured instances of {@link Kafka011TableSource}.
+ */
+public class Kafka011TableSourceSinkFactory extends KafkaTableSourceSinkFactoryBase {
+
+ @Override
+ protected String kafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_011;
+ }
+
+ @Override
+ protected boolean supportsKafkaTimestamps() {
+ return true;
+ }
+
+ @Override
+ protected KafkaTableSource createKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka011TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
+ }
+
+ @Override
+ protected KafkaTableSink createKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka011TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory b/flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
similarity index 91%
rename from flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
rename to flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
index 75135e57d74..b59b4a77ef5 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
+++ b/flink-connectors/flink-connector-kafka-0.11/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.flink.streaming.connectors.kafka.Kafka011JsonTableSourceFactory
+org.apache.flink.streaming.connectors.kafka.Kafka011TableSourceSinkFactory
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011ITCase.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011ITCase.java
index 361f2696919..74c58ad2891 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011ITCase.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011ITCase.java
@@ -24,6 +24,7 @@
import org.apache.flink.api.common.typeutils.base.IntSerializer;
import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
import org.apache.flink.runtime.checkpoint.StateObjectCollection;
+import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.state.OperatorStateHandle;
import org.apache.flink.streaming.api.operators.StreamSink;
import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
@@ -46,9 +47,12 @@
import java.util.stream.Collectors;
import java.util.stream.IntStream;
+import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.Semantic;
import static org.apache.flink.util.ExceptionUtils.findThrowable;
import static org.apache.flink.util.Preconditions.checkState;
+import static org.hamcrest.Matchers.lessThan;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
/**
* IT cases for the {@link FlinkKafkaProducer011}.
@@ -76,6 +80,43 @@ public void before() {
extraProperties.put("isolation.level", "read_committed");
}
+ @Test
+ public void resourceCleanUpNone() throws Exception {
+ resourceCleanUp(Semantic.NONE);
+ }
+
+ @Test
+ public void resourceCleanUpAtLeastOnce() throws Exception {
+ resourceCleanUp(Semantic.AT_LEAST_ONCE);
+ }
+
+ /**
+ * This tests checks whether there is some resource leak in form of growing threads number.
+ */
+ public void resourceCleanUp(Semantic semantic) throws Exception {
+ String topic = "flink-kafka-producer-resource-cleanup-" + semantic;
+
+ final int allowedEpsilonThreadCountGrow = 50;
+
+ Optional<Integer> initialActiveThreads = Optional.empty();
+ for (int i = 0; i < allowedEpsilonThreadCountGrow * 2; i++) {
+ try (OneInputStreamOperatorTestHarness<Integer, Object> testHarness1 =
+ createTestHarness(topic, 1, 1, 0, semantic)) {
+ testHarness1.setup();
+ testHarness1.open();
+ }
+
+ if (initialActiveThreads.isPresent()) {
+ assertThat("active threads count",
+ Thread.activeCount(),
+ lessThan(initialActiveThreads.get() + allowedEpsilonThreadCountGrow));
+ }
+ else {
+ initialActiveThreads = Optional.of(Thread.activeCount());
+ }
+ }
+ }
+
/**
* This test ensures that transactions reusing transactional.ids (after returning to the pool) will not clash
* with previous transactions using same transactional.ids.
@@ -176,7 +217,7 @@ public void testFlinkKafkaProducer011FailTransactionCoordinatorBeforeNotify() th
topic,
integerKeyedSerializationSchema,
properties,
- FlinkKafkaProducer011.Semantic.EXACTLY_ONCE);
+ Semantic.EXACTLY_ONCE);
OneInputStreamOperatorTestHarness<Integer, Object> testHarness1 = new OneInputStreamOperatorTestHarness<>(
new StreamSink<>(kafkaProducer),
@@ -327,7 +368,8 @@ public void testScaleDownBeforeFirstCheckpoint() throws Exception {
topic,
preScaleDownParallelism,
preScaleDownParallelism,
- subtaskIndex);
+ subtaskIndex,
+ Semantic.EXACTLY_ONCE);
preScaleDownOperator.setup();
preScaleDownOperator.open();
@@ -342,7 +384,7 @@ public void testScaleDownBeforeFirstCheckpoint() throws Exception {
// there might not be any close)
// After previous failure simulate restarting application with smaller parallelism
- OneInputStreamOperatorTestHarness<Integer, Object> postScaleDownOperator1 = createTestHarness(topic, 1, 1, 0);
+ OneInputStreamOperatorTestHarness<Integer, Object> postScaleDownOperator1 = createTestHarness(topic, 1, 1, 0, Semantic.EXACTLY_ONCE);
postScaleDownOperator1.setup();
postScaleDownOperator1.open();
@@ -443,7 +485,7 @@ public void testScaleUpAfterScalingDown() throws Exception {
for (int subtaskIndex = 0; subtaskIndex < parallelism; subtaskIndex++) {
OneInputStreamOperatorTestHarness<Integer, Object> testHarness =
- createTestHarness(topic, maxParallelism, parallelism, subtaskIndex);
+ createTestHarness(topic, maxParallelism, parallelism, subtaskIndex, Semantic.EXACTLY_ONCE);
testHarnesses.add(testHarness);
testHarness.setup();
@@ -564,28 +606,30 @@ private void closeIgnoringProducerFenced(AutoCloseable autoCloseable) throws Exc
}
private OneInputStreamOperatorTestHarness<Integer, Object> createTestHarness(String topic) throws Exception {
- return createTestHarness(topic, 1, 1, 0);
+ return createTestHarness(topic, 1, 1, 0, Semantic.EXACTLY_ONCE);
}
private OneInputStreamOperatorTestHarness<Integer, Object> createTestHarness(
- String topic,
- int maxParallelism,
- int parallelism,
- int subtaskIndex) throws Exception {
+ String topic,
+ int maxParallelism,
+ int parallelism,
+ int subtaskIndex,
+ Semantic semantic) throws Exception {
Properties properties = createProperties();
FlinkKafkaProducer011<Integer> kafkaProducer = new FlinkKafkaProducer011<>(
topic,
integerKeyedSerializationSchema,
properties,
- FlinkKafkaProducer011.Semantic.EXACTLY_ONCE);
+ semantic);
return new OneInputStreamOperatorTestHarness<>(
new StreamSink<>(kafkaProducer),
maxParallelism,
parallelism,
subtaskIndex,
- IntSerializer.INSTANCE);
+ IntSerializer.INSTANCE,
+ new OperatorID(42, 44));
}
private Properties createProperties() {
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSourceTest.java
index e348aa6bfb3..aa083a20535 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011AvroTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka011AvroTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka011AvroTableSourceTest extends KafkaAvroTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ITCase.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ITCase.java
index f48f87a773f..8692b5a6963 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ITCase.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ITCase.java
@@ -21,9 +21,9 @@
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.serialization.TypeInformationSerializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.java.typeutils.GenericTypeInfo;
-import org.apache.flink.api.java.typeutils.TypeInfoParser;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
import org.apache.flink.streaming.api.TimeCharacteristic;
@@ -213,7 +213,7 @@ public void cancel() {
}
});
- final TypeInformationSerializationSchema<Long> longSer = new TypeInformationSerializationSchema<>(TypeInfoParser.<Long>parse("Long"), env.getConfig());
+ final TypeInformationSerializationSchema<Long> longSer = new TypeInformationSerializationSchema<>(Types.LONG, env.getConfig());
FlinkKafkaProducer011<Long> prod = new FlinkKafkaProducer011<>(topic, new KeyedSerializationSchemaWrapper<>(longSer), standardProps, Optional.of(new FlinkKafkaPartitioner<Long>() {
private static final long serialVersionUID = -6730989584364230617L;
@@ -327,7 +327,7 @@ public void close() throws Exception {
long cnt = 0;
public LimitedLongDeserializer() {
- this.ti = TypeInfoParser.parse("Long");
+ this.ti = Types.LONG;
this.ser = ti.createSerializer(new ExecutionConfig());
}
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceFactoryTest.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceFactoryTest.java
index ed92863ba6d..ae46dac6a6a 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceFactoryTest.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceFactoryTest.java
@@ -21,8 +21,12 @@
import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_VERSION_VALUE_011;
/**
- * Tests for {@link Kafka011JsonTableSourceFactory}.
+ * Tests for legacy Kafka011JsonTableSourceFactory.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka011JsonTableSourceFactoryTest extends KafkaJsonTableSourceFactoryTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceTest.java
index b35d1c55072..451795a5bd5 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011JsonTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka011JsonTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka011JsonTableSourceTest extends KafkaJsonTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ProducerExactlyOnceITCase.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ProducerExactlyOnceITCase.java
index 116723869f4..5038b7f58f4 100644
--- a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ProducerExactlyOnceITCase.java
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ProducerExactlyOnceITCase.java
@@ -19,6 +19,7 @@
package org.apache.flink.streaming.connectors.kafka;
import org.junit.BeforeClass;
+import org.junit.Test;
/**
* IT cases for the {@link FlinkKafkaProducer011}.
@@ -48,4 +49,9 @@ public void testOneToOneAtLeastOnceCustomOperator() throws Exception {
// that it doesn't work well with some weird network failures, or the NetworkFailureProxy is a broken design
// and this test should be reimplemented in completely different way...
}
+
+ @Test
+ public void testMultipleSinkOperators() throws Exception {
+ testExactlyOnce(false, 2);
+ }
}
diff --git a/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactoryTest.java b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactoryTest.java
new file mode 100644
index 00000000000..996c5083860
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011TableSourceSinkFactoryTest.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Test for {@link Kafka011TableSource} and {@link Kafka011TableSink} created
+ * by {@link Kafka011TableSourceSinkFactory}.
+ */
+public class Kafka011TableSourceSinkFactoryTest extends KafkaTableSourceSinkFactoryTestBase {
+
+ @Override
+ protected String getKafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_011;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ protected Class<FlinkKafkaConsumerBase<Row>> getExpectedFlinkKafkaConsumer() {
+ return (Class) FlinkKafkaConsumer011.class;
+ }
+
+ @Override
+ protected Class<?> getExpectedFlinkKafkaProducer() {
+ return FlinkKafkaProducer011.class;
+ }
+
+ @Override
+ protected KafkaTableSource getExpectedKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka011TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets
+ );
+ }
+
+ @Override
+ protected KafkaTableSink getExpectedKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka011TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema
+ );
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.8/pom.xml b/flink-connectors/flink-connector-kafka-0.8/pom.xml
index 7750ef37e7b..df1afb57aa5 100644
--- a/flink-connectors/flink-connector-kafka-0.8/pom.xml
+++ b/flink-connectors/flink-connector-kafka-0.8/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSource.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSource.java
index 3c16722add6..61c96bf1e5a 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -35,8 +35,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.8.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka08AvroTableSource extends KafkaAvroTableSource {
/**
@@ -46,7 +51,9 @@
* @param properties Properties for the Kafka consumer.
* @param schema Schema of the produced table.
* @param record Avro specific record.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka08AvroTableSource(
String topic,
Properties properties,
@@ -69,7 +76,9 @@ public Kafka08AvroTableSource(
* the value to the field of the Avro record.</p>
*
* @param fieldMapping A mapping from schema fields to Avro fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -79,7 +88,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -89,7 +100,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -102,15 +115,27 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka08AvroTableSource}.
+ *
* @return A builder to configure and create a {@link Kafka08AvroTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Builder builder() {
return new Builder();
}
/**
* A builder to configure and create a {@link Kafka08AvroTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaAvroTableSource.Builder<Kafka08AvroTableSource, Kafka08AvroTableSource.Builder> {
@Override
@@ -127,7 +152,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka08AvroTableSource}.
*
* @return A configured {@link Kafka08AvroTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka08AvroTableSource build() {
Kafka08AvroTableSource tableSource = new Kafka08AvroTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSink.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSink.java
index c60288d3f01..45588cdb141 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSink.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSink.java
@@ -18,20 +18,25 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaDelegatePartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.types.Row;
import java.util.Properties;
/**
* Kafka 0.8 {@link KafkaTableSink} that serializes data in JSON format.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka08JsonTableSink extends KafkaJsonTableSink {
/**
@@ -48,7 +53,9 @@
*
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka08JsonTableSink(String topic, Properties properties) {
super(topic, properties, new FlinkFixedPartitioner<>());
}
@@ -60,7 +67,9 @@ public Kafka08JsonTableSink(String topic, Properties properties) {
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
* @param partitioner Kafka partitioner
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka08JsonTableSink(String topic, Properties properties, FlinkKafkaPartitioner<Row> partitioner) {
super(topic, properties, partitioner);
}
@@ -84,7 +93,11 @@ public Kafka08JsonTableSink(String topic, Properties properties, KafkaPartitione
@Override
protected FlinkKafkaProducerBase<Row> createKafkaProducer(String topic, Properties properties, SerializationSchema<Row> serializationSchema, FlinkKafkaPartitioner<Row> partitioner) {
- return new FlinkKafkaProducer08<>(topic, serializationSchema, properties, partitioner);
+ return new FlinkKafkaProducer08<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSource.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSource.java
index 73870560fb3..dc5a077a89c 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -32,8 +32,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.8.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka08JsonTableSource extends KafkaJsonTableSource {
/**
@@ -43,7 +48,9 @@
* @param properties Properties for the Kafka consumer.
* @param tableSchema The schema of the table.
* @param jsonSchema The schema of the JSON messages to decode from Kafka.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka08JsonTableSource(
String topic,
Properties properties,
@@ -58,7 +65,9 @@ public Kafka08JsonTableSource(
* TableSource will fail for missing fields if set to true. If set to false, the missing field is set to null.
*
* @param failOnMissingField Flag that specifies the TableSource behavior in case of missing fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFailOnMissingField(boolean failOnMissingField) {
super.setFailOnMissingField(failOnMissingField);
@@ -68,7 +77,9 @@ public void setFailOnMissingField(boolean failOnMissingField) {
* Sets the mapping from table schema fields to JSON schema fields.
*
* @param fieldMapping The mapping from table schema fields to JSON schema fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -78,7 +89,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -88,7 +101,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -102,14 +117,25 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka08JsonTableSource}.
* @return A builder to configure and create a {@link Kafka08JsonTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Kafka08JsonTableSource.Builder builder() {
return new Kafka08JsonTableSource.Builder();
}
/**
* A builder to configure and create a {@link Kafka08JsonTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaJsonTableSource.Builder<Kafka08JsonTableSource, Kafka08JsonTableSource.Builder> {
@Override
@@ -126,7 +152,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka08JsonTableSource}.
*
* @return A configured {@link Kafka08JsonTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka08JsonTableSource build() {
Kafka08JsonTableSource tableSource = new Kafka08JsonTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSink.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSink.java
new file mode 100644
index 00000000000..c34de13efde
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSink.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+
+import java.util.Properties;
+
+/**
+ * Kafka 0.8 table sink for writing data into Kafka.
+ */
+@Internal
+public class Kafka08TableSink extends KafkaTableSink {
+
+ public Kafka08TableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+ super(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+
+ @Override
+ protected FlinkKafkaProducerBase<Row> createKafkaProducer(
+ String topic,
+ Properties properties,
+ SerializationSchema<Row> serializationSchema,
+ FlinkKafkaPartitioner<Row> partitioner) {
+ return new FlinkKafkaProducer08<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSource.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSource.java
index 918964d67f4..97c293e0690 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSource.java
@@ -18,48 +18,79 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
import java.util.Properties;
/**
* Kafka {@link StreamTableSource} for Kafka 0.8.
*/
-@PublicEvolving
-public abstract class Kafka08TableSource extends KafkaTableSource {
-
- // The deserialization schema for the Kafka records
- private final DeserializationSchema<Row> deserializationSchema;
+@Internal
+public class Kafka08TableSource extends KafkaTableSource {
/**
* Creates a Kafka 0.8 {@link StreamTableSource}.
*
- * @param topic Kafka topic to consume.
- * @param properties Properties for the Kafka consumer.
- * @param deserializationSchema Deserialization schema to use for Kafka records.
- * @param typeInfo Type information describing the result type. The field names are used
- * to parse the JSON file and so are the types.
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*/
public Kafka08TableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Optional<Map<String, String>> fieldMapping,
String topic,
Properties properties,
DeserializationSchema<Row> deserializationSchema,
- TableSchema schema,
- TypeInformation<Row> typeInfo) {
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
- super(topic, properties, schema, typeInfo);
-
- this.deserializationSchema = deserializationSchema;
+ super(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ fieldMapping,
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
}
- @Override
- public DeserializationSchema<Row> getDeserializationSchema() {
- return this.deserializationSchema;
+ /**
+ * Creates a Kafka 0.8 {@link StreamTableSource}.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ */
+ public Kafka08TableSource(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema) {
+
+ super(schema, topic, properties, deserializationSchema);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactory.java b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactory.java
new file mode 100644
index 00000000000..3e93b6fdeac
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Factory for creating configured instances of {@link Kafka08TableSource}.
+ */
+public class Kafka08TableSourceSinkFactory extends KafkaTableSourceSinkFactoryBase {
+
+ @Override
+ protected String kafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_08;
+ }
+
+ @Override
+ protected boolean supportsKafkaTimestamps() {
+ return false;
+ }
+
+ @Override
+ protected KafkaTableSource createKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka08TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
+ }
+
+ @Override
+ protected KafkaTableSink createKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka08TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory b/flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
similarity index 91%
rename from flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
rename to flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
index 2f38bd0e974..f2e1c3f5e60 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
+++ b/flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.flink.streaming.connectors.kafka.Kafka09JsonTableSourceFactory
+org.apache.flink.streaming.connectors.kafka.Kafka08TableSourceSinkFactory
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSourceTest.java
index 22daafe8e1a..c7b8e8c91d1 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08AvroTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka08AvroTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka08AvroTableSourceTest extends KafkaAvroTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSinkTest.java b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSinkTest.java
index 53da9f6ba49..32bd3b69c06 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSinkTest.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSinkTest.java
@@ -27,7 +27,11 @@
/**
* Tests for the {@link Kafka08JsonTableSink}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sinks.
*/
+@Deprecated
public class Kafka08JsonTableSinkTest extends KafkaTableSinkTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceFactoryTest.java b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceFactoryTest.java
index 0238b2bf4f9..915ce42d313 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceFactoryTest.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceFactoryTest.java
@@ -21,8 +21,12 @@
import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_VERSION_VALUE_08;
/**
- * Tests for {@link Kafka08JsonTableSourceFactory}.
+ * Tests for legacy Kafka08JsonTableSourceFactory.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka08JsonTableSourceFactoryTest extends KafkaJsonTableSourceFactoryTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceTest.java
index b66e8ffc9ce..4ce79ddb8ee 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08JsonTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka08JsonTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka08JsonTableSourceTest extends KafkaJsonTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactoryTest.java b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactoryTest.java
new file mode 100644
index 00000000000..b67501e449e
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka08TableSourceSinkFactoryTest.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Test for {@link Kafka08TableSource} and {@link Kafka08TableSink} created
+ * by {@link Kafka08TableSourceSinkFactory}.
+ */
+public class Kafka08TableSourceSinkFactoryTest extends KafkaTableSourceSinkFactoryTestBase {
+
+ @Override
+ protected String getKafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_08;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ protected Class<FlinkKafkaConsumerBase<Row>> getExpectedFlinkKafkaConsumer() {
+ return (Class) FlinkKafkaConsumer08.class;
+ }
+
+ @Override
+ protected Class<?> getExpectedFlinkKafkaProducer() {
+ return FlinkKafkaProducer08.class;
+ }
+
+ @Override
+ protected KafkaTableSource getExpectedKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka08TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets
+ );
+ }
+
+ @Override
+ protected KafkaTableSink getExpectedKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka08TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema
+ );
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.9/pom.xml b/flink-connectors/flink-connector-kafka-0.9/pom.xml
index a4b0cfc3c1e..3e0aa46e095 100644
--- a/flink-connectors/flink-connector-kafka-0.9/pom.xml
+++ b/flink-connectors/flink-connector-kafka-0.9/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
@@ -190,6 +190,58 @@ under the License.
</dependencies>
+ <profiles>
+ <!-- Create SQL Client uber jars by default -->
+ <profile>
+ <id>sql-jars</id>
+ <activation>
+ <property>
+ <name>!skipSqlJars</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadedArtifactAttached>true</shadedArtifactAttached>
+ <shadedClassifierName>sql-jar</shadedClassifierName>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.kafka:*</include>
+ <include>org.apache.flink:flink-connector-kafka-base_${scala.binary.version}</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>kafka/kafka-version.properties</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+ <pattern>org.apache.kafka</pattern>
+ <shadedPattern>org.apache.flink.kafka09.shaded.org.apache.kafka</shadedPattern>
+ </relocation>
+ </relocations>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
<build>
<plugins>
<plugin>
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSource.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSource.java
index 7edbe651059..4352d7e400d 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -35,8 +35,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.9.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka09AvroTableSource extends KafkaAvroTableSource {
/**
@@ -69,7 +74,9 @@ public Kafka09AvroTableSource(
* the value to the field of the Avro record.</p>
*
* @param fieldMapping A mapping from schema fields to Avro fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -79,7 +86,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -89,7 +98,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -102,15 +113,27 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka09AvroTableSource}.
+ *
* @return A builder to configure and create a {@link Kafka09AvroTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Builder builder() {
return new Builder();
}
/**
* A builder to configure and create a {@link Kafka09AvroTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaAvroTableSource.Builder<Kafka09AvroTableSource, Kafka09AvroTableSource.Builder> {
@Override
@@ -127,7 +150,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka09AvroTableSource}.
*
* @return A configured {@link Kafka09AvroTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka09AvroTableSource build() {
Kafka09AvroTableSource tableSource = new Kafka09AvroTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.java
index 95ce4e6eff3..b3cc0aa77ca 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.java
@@ -18,20 +18,25 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaDelegatePartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.types.Row;
import java.util.Properties;
/**
* Kafka 0.9 {@link KafkaTableSink} that serializes data in JSON format.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka09JsonTableSink extends KafkaJsonTableSink {
/**
@@ -48,7 +53,9 @@
*
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka09JsonTableSink(String topic, Properties properties) {
super(topic, properties, new FlinkFixedPartitioner<>());
}
@@ -60,7 +67,9 @@ public Kafka09JsonTableSink(String topic, Properties properties) {
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
* @param partitioner Kafka partitioner
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public Kafka09JsonTableSink(String topic, Properties properties, FlinkKafkaPartitioner<Row> partitioner) {
super(topic, properties, partitioner);
}
@@ -84,7 +93,11 @@ public Kafka09JsonTableSink(String topic, Properties properties, KafkaPartitione
@Override
protected FlinkKafkaProducerBase<Row> createKafkaProducer(String topic, Properties properties, SerializationSchema<Row> serializationSchema, FlinkKafkaPartitioner<Row> partitioner) {
- return new FlinkKafkaProducer09<>(topic, serializationSchema, properties, partitioner);
+ return new FlinkKafkaProducer09<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.java
index a7e54be2275..db1df3d889e 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.java
@@ -18,9 +18,9 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
@@ -32,8 +32,13 @@
/**
* Kafka {@link StreamTableSource} for Kafka 0.9.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
-@PublicEvolving
+@Deprecated
public class Kafka09JsonTableSource extends KafkaJsonTableSource {
/**
@@ -43,7 +48,9 @@
* @param properties Properties for the Kafka consumer.
* @param tableSchema The schema of the table.
* @param jsonSchema The schema of the JSON messages to decode from Kafka.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public Kafka09JsonTableSource(
String topic,
Properties properties,
@@ -58,7 +65,9 @@ public Kafka09JsonTableSource(
* TableSource will fail for missing fields if set to true. If set to false, the missing field is set to null.
*
* @param failOnMissingField Flag that specifies the TableSource behavior in case of missing fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFailOnMissingField(boolean failOnMissingField) {
super.setFailOnMissingField(failOnMissingField);
@@ -68,7 +77,9 @@ public void setFailOnMissingField(boolean failOnMissingField) {
* Sets the mapping from table schema fields to JSON schema fields.
*
* @param fieldMapping The mapping from table schema fields to JSON schema fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setFieldMapping(Map<String, String> fieldMapping) {
super.setFieldMapping(fieldMapping);
@@ -78,7 +89,9 @@ public void setFieldMapping(Map<String, String> fieldMapping) {
* Declares a field of the schema to be a processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public void setProctimeAttribute(String proctimeAttribute) {
super.setProctimeAttribute(proctimeAttribute);
@@ -88,7 +101,9 @@ public void setProctimeAttribute(String proctimeAttribute) {
* Declares a field of the schema to be a rowtime attribute.
*
* @param rowtimeAttributeDescriptor The descriptor of the rowtime attribute.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttributeDescriptor) {
Preconditions.checkNotNull(rowtimeAttributeDescriptor, "Rowtime attribute descriptor must not be null.");
super.setRowtimeAttributeDescriptors(Collections.singletonList(rowtimeAttributeDescriptor));
@@ -102,14 +117,25 @@ public void setRowtimeAttributeDescriptor(RowtimeAttributeDescriptor rowtimeAttr
/**
* Returns a builder to configure and create a {@link Kafka09JsonTableSource}.
* @return A builder to configure and create a {@link Kafka09JsonTableSource}.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static Kafka09JsonTableSource.Builder builder() {
return new Kafka09JsonTableSource.Builder();
}
/**
* A builder to configure and create a {@link Kafka09JsonTableSource}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
public static class Builder extends KafkaJsonTableSource.Builder<Kafka09JsonTableSource, Kafka09JsonTableSource.Builder> {
@Override
@@ -126,7 +152,9 @@ protected boolean supportsKafkaTimestamps() {
* Builds and configures a {@link Kafka09JsonTableSource}.
*
* @return A configured {@link Kafka09JsonTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
@Override
public Kafka09JsonTableSource build() {
Kafka09JsonTableSource tableSource = new Kafka09JsonTableSource(
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSink.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSink.java
new file mode 100644
index 00000000000..8c349d7a0b2
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSink.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+
+import java.util.Properties;
+
+/**
+ * Kafka 0.9 table sink for writing data into Kafka.
+ */
+@Internal
+public class Kafka09TableSink extends KafkaTableSink {
+
+ public Kafka09TableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+ super(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+
+ @Override
+ protected FlinkKafkaProducerBase<Row> createKafkaProducer(
+ String topic,
+ Properties properties,
+ SerializationSchema<Row> serializationSchema,
+ FlinkKafkaPartitioner<Row> partitioner) {
+ return new FlinkKafkaProducer09<>(
+ topic,
+ serializationSchema,
+ properties,
+ partitioner);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.java
index 31518de5c32..8f9e799ec96 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.java
@@ -18,48 +18,79 @@
package org.apache.flink.streaming.connectors.kafka;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import org.apache.flink.types.Row;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
import java.util.Properties;
/**
* Kafka {@link StreamTableSource} for Kafka 0.9.
*/
-@PublicEvolving
-public abstract class Kafka09TableSource extends KafkaTableSource {
-
- // The deserialization schema for the Kafka records
- private final DeserializationSchema<Row> deserializationSchema;
+@Internal
+public class Kafka09TableSource extends KafkaTableSource {
/**
* Creates a Kafka 0.9 {@link StreamTableSource}.
*
- * @param topic Kafka topic to consume.
- * @param properties Properties for the Kafka consumer.
- * @param deserializationSchema Deserialization schema to use for Kafka records.
- * @param typeInfo Type information describing the result type. The field names are used
- * to parse the JSON file and so are the types.
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*/
public Kafka09TableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Optional<Map<String, String>> fieldMapping,
String topic,
Properties properties,
DeserializationSchema<Row> deserializationSchema,
- TableSchema schema,
- TypeInformation<Row> typeInfo) {
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
- super(topic, properties, schema, typeInfo);
-
- this.deserializationSchema = deserializationSchema;
+ super(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ fieldMapping,
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
}
- @Override
- public DeserializationSchema<Row> getDeserializationSchema() {
- return this.deserializationSchema;
+ /**
+ * Creates a Kafka 0.9 {@link StreamTableSource}.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ */
+ public Kafka09TableSource(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema) {
+
+ super(schema, topic, properties, deserializationSchema);
}
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactory.java b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactory.java
new file mode 100644
index 00000000000..9958b4ef316
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Factory for creating configured instances of {@link Kafka09TableSource}.
+ */
+public class Kafka09TableSourceSinkFactory extends KafkaTableSourceSinkFactoryBase {
+
+ @Override
+ protected String kafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_09;
+ }
+
+ @Override
+ protected boolean supportsKafkaTimestamps() {
+ return false;
+ }
+
+ @Override
+ protected KafkaTableSource createKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka09TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
+ }
+
+ @Override
+ protected KafkaTableSink createKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka09TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema);
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory b/flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
similarity index 91%
rename from flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
rename to flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
index 9092955842e..26258736937 100644
--- a/flink-connectors/flink-connector-kafka-0.8/src/main/resources/META-INF/services/org.apache.flink.table.sources.TableSourceFactory
+++ b/flink-connectors/flink-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.flink.streaming.connectors.kafka.Kafka08JsonTableSourceFactory
+org.apache.flink.streaming.connectors.kafka.Kafka09TableSourceSinkFactory
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSourceTest.java
index 0ab8a4be61c..6f3a56639b3 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09AvroTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka09AvroTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka09AvroTableSourceTest extends KafkaAvroTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSinkTest.java b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSinkTest.java
index 610e048af2a..79f251b8302 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSinkTest.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSinkTest.java
@@ -27,7 +27,11 @@
/**
* Tests for the {@link Kafka09JsonTableSink}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sinks.
*/
+@Deprecated
public class Kafka09JsonTableSinkTest extends KafkaTableSinkTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceFactoryTest.java b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceFactoryTest.java
index dd545e93951..e0437a10c4e 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceFactoryTest.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceFactoryTest.java
@@ -22,7 +22,11 @@
/**
* Factory for creating configured instances of {@link Kafka09JsonTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka09JsonTableSourceFactoryTest extends KafkaJsonTableSourceFactoryTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceTest.java b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceTest.java
index 65fa0f1796b..bfff7d20102 100644
--- a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceTest.java
+++ b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSourceTest.java
@@ -24,7 +24,11 @@
/**
* Tests for the {@link Kafka09JsonTableSource}.
+ *
+ * @deprecated Ensures backwards compatibility with Flink 1.5. Can be removed once we
+ * drop support for format-specific table sources.
*/
+@Deprecated
public class Kafka09JsonTableSourceTest extends KafkaJsonTableSourceTestBase {
@Override
diff --git a/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactoryTest.java b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactoryTest.java
new file mode 100644
index 00000000000..a6c8bd4b279
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09TableSourceSinkFactoryTest.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.types.Row;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Test for {@link Kafka09TableSource} and {@link Kafka09TableSink} created
+ * by {@link Kafka09TableSourceSinkFactory}.
+ */
+public class Kafka09TableSourceSinkFactoryTest extends KafkaTableSourceSinkFactoryTestBase {
+
+ @Override
+ protected String getKafkaVersion() {
+ return KafkaValidator.CONNECTOR_VERSION_VALUE_09;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ protected Class<FlinkKafkaConsumerBase<Row>> getExpectedFlinkKafkaConsumer() {
+ return (Class) FlinkKafkaConsumer09.class;
+ }
+
+ @Override
+ protected Class<?> getExpectedFlinkKafkaProducer() {
+ return FlinkKafkaProducer09.class;
+ }
+
+ @Override
+ protected KafkaTableSource getExpectedKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+
+ return new Kafka09TableSource(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ Optional.of(fieldMapping),
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets
+ );
+ }
+
+ @Override
+ protected KafkaTableSink getExpectedKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+
+ return new Kafka09TableSink(
+ schema,
+ topic,
+ properties,
+ partitioner,
+ serializationSchema
+ );
+ }
+}
diff --git a/flink-connectors/flink-connector-kafka-base/pom.xml b/flink-connectors/flink-connector-kafka-base/pom.xml
index 1694afe8f24..d6a6bb6d5ea 100644
--- a/flink-connectors/flink-connector-kafka-base/pom.xml
+++ b/flink-connectors/flink-connector-kafka-base/pom.xml
@@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connectors</artifactId>
- <version>1.6-SNAPSHOT</version>
+ <version>1.7-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
index 82ac2c37b8f..cfb5b6d510d 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
@@ -715,7 +715,7 @@ public void run() {
}
}
}
- });
+ }, "Kafka Partition Discovery for " + getRuntimeContext().getTaskNameWithSubtasks());
discoveryLoopThread.start();
kafkaFetcher.runFetchLoop();
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaAvroTableSource.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaAvroTableSource.java
index 1587798be5b..86fd21d15c6 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaAvroTableSource.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaAvroTableSource.java
@@ -20,24 +20,14 @@
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.java.typeutils.GenericTypeInfo;
-import org.apache.flink.api.java.typeutils.RowTypeInfo;
import org.apache.flink.formats.avro.AvroRowDeserializationSchema;
-import org.apache.flink.formats.avro.typeutils.AvroTypeInfo;
import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.sources.DefinedFieldMapping;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
-import org.apache.flink.types.Row;
-import org.apache.avro.Schema;
-import org.apache.avro.specific.SpecificData;
import org.apache.avro.specific.SpecificRecord;
import org.apache.avro.specific.SpecificRecordBase;
-import org.apache.avro.util.Utf8;
-import java.util.List;
import java.util.Map;
import java.util.Properties;
@@ -46,13 +36,15 @@
*
* <p>The version-specific Kafka consumers need to extend this class and
* override {@link #createKafkaConsumer(String, Properties, DeserializationSchema)}}.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+@Deprecated
@Internal
-public abstract class KafkaAvroTableSource extends KafkaTableSource implements DefinedFieldMapping {
-
- private final Class<? extends SpecificRecordBase> avroRecordClass;
-
- private Map<String, String> fieldMapping;
+public abstract class KafkaAvroTableSource extends KafkaTableSource {
/**
* Creates a generic Kafka Avro {@link StreamTableSource} using a given {@link SpecificRecord}.
@@ -61,7 +53,12 @@
* @param properties Properties for the Kafka consumer.
* @param schema Schema of the produced table.
* @param avroRecordClass Class of the Avro record that is read from the Kafka topic.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
protected KafkaAvroTableSource(
String topic,
Properties properties,
@@ -69,87 +66,32 @@ protected KafkaAvroTableSource(
Class<? extends SpecificRecordBase> avroRecordClass) {
super(
+ schema,
topic,
properties,
- schema,
- convertToRowTypeInformation(avroRecordClass));
-
- this.avroRecordClass = avroRecordClass;
- }
-
- @Override
- public Map<String, String> getFieldMapping() {
- return fieldMapping;
+ new AvroRowDeserializationSchema(avroRecordClass));
}
@Override
public String explainSource() {
- return "KafkaAvroTableSource(" + this.avroRecordClass.getSimpleName() + ")";
- }
-
- @Override
- protected AvroRowDeserializationSchema getDeserializationSchema() {
- return new AvroRowDeserializationSchema(avroRecordClass);
- }
-
- //////// SETTERS FOR OPTIONAL PARAMETERS
-
- /**
- * Configures a field mapping for this TableSource.
- *
- * @param fieldMapping The field mapping.
- */
- protected void setFieldMapping(Map<String, String> fieldMapping) {
- this.fieldMapping = fieldMapping;
+ return "KafkaAvroTableSource";
}
//////// HELPER METHODS
- /**
- * Converts the extracted AvroTypeInfo into a RowTypeInfo nested structure with deterministic field order.
- * Replaces generic Utf8 with basic String type information.
- */
- @SuppressWarnings("unchecked")
- private static <T extends SpecificRecordBase> TypeInformation<Row> convertToRowTypeInformation(Class<T> avroClass) {
- final AvroTypeInfo<T> avroTypeInfo = new AvroTypeInfo<>(avroClass);
- // determine schema to retrieve deterministic field order
- final Schema schema = SpecificData.get().getSchema(avroClass);
- return (TypeInformation<Row>) convertToTypeInformation(avroTypeInfo, schema);
- }
-
- /**
- * Recursively converts extracted AvroTypeInfo into a RowTypeInfo nested structure with deterministic field order.
- * Replaces generic Utf8 with basic String type information.
- */
- private static TypeInformation<?> convertToTypeInformation(TypeInformation<?> extracted, Schema schema) {
- if (schema.getType() == Schema.Type.RECORD) {
- final List<Schema.Field> fields = schema.getFields();
- final AvroTypeInfo<?> avroTypeInfo = (AvroTypeInfo<?>) extracted;
-
- final TypeInformation<?>[] types = new TypeInformation<?>[fields.size()];
- final String[] names = new String[fields.size()];
- for (int i = 0; i < fields.size(); i++) {
- final Schema.Field field = fields.get(i);
- types[i] = convertToTypeInformation(avroTypeInfo.getTypeAt(field.name()), field.schema());
- names[i] = field.name();
- }
- return new RowTypeInfo(types, names);
- } else if (extracted instanceof GenericTypeInfo<?>) {
- final GenericTypeInfo<?> genericTypeInfo = (GenericTypeInfo<?>) extracted;
- if (genericTypeInfo.getTypeClass() == Utf8.class) {
- return BasicTypeInfo.STRING_TYPE_INFO;
- }
- }
- return extracted;
- }
-
/**
* Abstract builder for a {@link KafkaAvroTableSource} to be extended by builders of subclasses of
* KafkaAvroTableSource.
*
* @param <T> Type of the KafkaAvroTableSource produced by the builder.
* @param <B> Type of the KafkaAvroTableSource.Builder subclass.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
protected abstract static class Builder<T extends KafkaAvroTableSource, B extends KafkaAvroTableSource.Builder>
extends KafkaTableSource.Builder<T, B> {
@@ -162,7 +104,9 @@ protected void setFieldMapping(Map<String, String> fieldMapping) {
*
* @param avroClass The class of the Avro records that are read from the Kafka topic.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B forAvroRecordClass(Class<? extends SpecificRecordBase> avroClass) {
this.avroClass = avroClass;
return builder();
@@ -178,7 +122,9 @@ public B forAvroRecordClass(Class<? extends SpecificRecordBase> avroClass) {
*
* @param schemaToAvroMapping A mapping from schema fields to Avro fields.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withTableToAvroMapping(Map<String, String> schemaToAvroMapping) {
this.fieldMapping = schemaToAvroMapping;
return builder();
@@ -188,7 +134,9 @@ public B withTableToAvroMapping(Map<String, String> schemaToAvroMapping) {
* Returns the configured Avro class.
*
* @return The configured Avro class.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected Class<? extends SpecificRecordBase> getAvroRecordClass() {
return this.avroClass;
}
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSink.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSink.java
index ec273989a55..231edddb311 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSink.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSink.java
@@ -29,7 +29,10 @@
/**
* Base class for {@link KafkaTableSink} that serializes data in JSON format.
+ *
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+@Deprecated
@Internal
public abstract class KafkaJsonTableSink extends KafkaTableSink {
@@ -39,7 +42,9 @@
* @param topic topic in Kafka to which table is written
* @param properties properties to connect to Kafka
* @param partitioner Kafka partitioner
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public KafkaJsonTableSink(String topic, Properties properties, FlinkKafkaPartitioner<Row> partitioner) {
super(topic, properties, partitioner);
}
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSource.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSource.java
index b2bb8ff773b..70b286bc296 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSource.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSource.java
@@ -20,14 +20,12 @@
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.java.typeutils.RowTypeInfo;
import org.apache.flink.formats.json.JsonRowDeserializationSchema;
import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.sources.DefinedFieldMapping;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
import org.apache.flink.table.sources.StreamTableSource;
import java.util.Map;
-import java.util.Objects;
import java.util.Properties;
/**
@@ -37,15 +35,15 @@
* override {@link #createKafkaConsumer(String, Properties, DeserializationSchema)}}.
*
* <p>The field names are used to parse the JSON file and so are the types.
+ *
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+@Deprecated
@Internal
-public abstract class KafkaJsonTableSource extends KafkaTableSource implements DefinedFieldMapping {
-
- private TableSchema jsonSchema;
-
- private Map<String, String> fieldMapping;
-
- private boolean failOnMissingField;
+public abstract class KafkaJsonTableSource extends KafkaTableSource {
/**
* Creates a generic Kafka JSON {@link StreamTableSource}.
@@ -54,7 +52,9 @@
* @param properties Properties for the Kafka consumer.
* @param tableSchema The schema of the table.
* @param jsonSchema The schema of the JSON messages to decode from Kafka.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected KafkaJsonTableSource(
String topic,
Properties properties,
@@ -62,24 +62,10 @@ protected KafkaJsonTableSource(
TableSchema jsonSchema) {
super(
+ tableSchema,
topic,
properties,
- tableSchema,
- jsonSchemaToReturnType(jsonSchema));
-
- this.jsonSchema = jsonSchema;
- }
-
- @Override
- public Map<String, String> getFieldMapping() {
- return fieldMapping;
- }
-
- @Override
- protected JsonRowDeserializationSchema getDeserializationSchema() {
- JsonRowDeserializationSchema deserSchema = new JsonRowDeserializationSchema(jsonSchemaToReturnType(jsonSchema));
- deserSchema.setFailOnMissingField(failOnMissingField);
- return deserSchema;
+ new JsonRowDeserializationSchema(jsonSchema.toRowType()));
}
@Override
@@ -87,28 +73,6 @@ public String explainSource() {
return "KafkaJsonTableSource";
}
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof KafkaJsonTableSource)) {
- return false;
- }
- if (!super.equals(o)) {
- return false;
- }
- KafkaJsonTableSource that = (KafkaJsonTableSource) o;
- return failOnMissingField == that.failOnMissingField &&
- Objects.equals(jsonSchema, that.jsonSchema) &&
- Objects.equals(fieldMapping, that.fieldMapping);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(super.hashCode(), jsonSchema, fieldMapping, failOnMissingField);
- }
-
//////// SETTERS FOR OPTIONAL PARAMETERS
/**
@@ -116,34 +80,27 @@ public int hashCode() {
* TableSource will fail for missing fields if set to true. If set to false, the missing field is set to null.
*
* @param failOnMissingField Flag that specifies the TableSource behavior in case of missing fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected void setFailOnMissingField(boolean failOnMissingField) {
- this.failOnMissingField = failOnMissingField;
- }
-
- /**
- * Sets the mapping from table schema fields to JSON schema fields.
- *
- * @param fieldMapping The mapping from table schema fields to JSON schema fields.
- */
- protected void setFieldMapping(Map<String, String> fieldMapping) {
- this.fieldMapping = fieldMapping;
+ ((JsonRowDeserializationSchema) getDeserializationSchema()).setFailOnMissingField(failOnMissingField);
}
//////// HELPER METHODS
- /** Converts the JSON schema into into the return type. */
- private static RowTypeInfo jsonSchemaToReturnType(TableSchema jsonSchema) {
- return new RowTypeInfo(jsonSchema.getTypes(), jsonSchema.getColumnNames());
- }
-
/**
* Abstract builder for a {@link KafkaJsonTableSource} to be extended by builders of subclasses of
* KafkaJsonTableSource.
*
* @param <T> Type of the KafkaJsonTableSource produced by the builder.
* @param <B> Type of the KafkaJsonTableSource.Builder subclass.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
protected abstract static class Builder<T extends KafkaJsonTableSource, B extends KafkaJsonTableSource.Builder>
extends KafkaTableSource.Builder<T, B> {
@@ -159,7 +116,9 @@ private static RowTypeInfo jsonSchemaToReturnType(TableSchema jsonSchema) {
*
* @param jsonSchema The schema of the JSON-encoded Kafka messages.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B forJsonSchema(TableSchema jsonSchema) {
this.jsonSchema = jsonSchema;
return builder();
@@ -175,7 +134,9 @@ public B forJsonSchema(TableSchema jsonSchema) {
*
* @param tableToJsonMapping A mapping from table schema fields to JSON schema fields.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withTableToJsonMapping(Map<String, String> tableToJsonMapping) {
this.fieldMapping = tableToJsonMapping;
return builder();
@@ -188,7 +149,9 @@ public B withTableToJsonMapping(Map<String, String> tableToJsonMapping) {
* field.
* If set to false, a missing field is set to null.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B failOnMissingField(boolean failOnMissingField) {
this.failOnMissingField = failOnMissingField;
return builder();
@@ -199,7 +162,9 @@ public B failOnMissingField(boolean failOnMissingField) {
* is returned.
*
* @return The JSON schema for the TableSource.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected TableSchema getJsonSchema() {
if (jsonSchema != null) {
return this.jsonSchema;
@@ -208,7 +173,13 @@ protected TableSchema getJsonSchema() {
}
}
- @Override
+ /**
+ * Configures a TableSource with optional parameters.
+ *
+ * @param source The TableSource to configure.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
+ */
+ @Deprecated
protected void configureTableSource(T source) {
super.configureTableSource(source);
// configure field mapping
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSourceFactory.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSourceFactory.java
deleted file mode 100644
index 28973149abd..00000000000
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSourceFactory.java
+++ /dev/null
@@ -1,238 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.streaming.connectors.kafka;
-
-import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.formats.json.JsonSchemaConverter;
-import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
-import org.apache.flink.table.api.TableException;
-import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.descriptors.DescriptorProperties;
-import org.apache.flink.table.descriptors.JsonValidator;
-import org.apache.flink.table.descriptors.KafkaValidator;
-import org.apache.flink.table.descriptors.SchemaValidator;
-import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
-import org.apache.flink.table.sources.TableSource;
-import org.apache.flink.table.sources.TableSourceFactory;
-import org.apache.flink.types.Row;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-import java.util.Properties;
-
-import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_PROPERTY_VERSION;
-import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_TYPE;
-import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_VERSION;
-import static org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT_DERIVE_SCHEMA;
-import static org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT_PROPERTY_VERSION;
-import static org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT_TYPE;
-import static org.apache.flink.table.descriptors.JsonValidator.FORMAT_FAIL_ON_MISSING_FIELD;
-import static org.apache.flink.table.descriptors.JsonValidator.FORMAT_JSON_SCHEMA;
-import static org.apache.flink.table.descriptors.JsonValidator.FORMAT_SCHEMA;
-import static org.apache.flink.table.descriptors.JsonValidator.FORMAT_TYPE_VALUE;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES_KEY;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES_VALUE;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS_OFFSET;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS_PARTITION;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_STARTUP_MODE;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_TOPIC;
-import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_TYPE_VALUE_KAFKA;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_CLASS;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_FROM;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_SERIALIZED;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_TYPE;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_CLASS;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_DELAY;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_SERIALIZED;
-import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_TYPE;
-import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA;
-import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_FROM;
-import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_NAME;
-import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_PROCTIME;
-import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_TYPE;
-
-/**
- * Factory for creating configured instances of {@link KafkaJsonTableSource}.
- */
-public abstract class KafkaJsonTableSourceFactory implements TableSourceFactory<Row> {
-
- @Override
- public Map<String, String> requiredContext() {
- Map<String, String> context = new HashMap<>();
- context.put(CONNECTOR_TYPE(), CONNECTOR_TYPE_VALUE_KAFKA); // kafka
- context.put(CONNECTOR_VERSION(), kafkaVersion());
-
- context.put(FORMAT_TYPE(), FORMAT_TYPE_VALUE); // json format
-
- context.put(CONNECTOR_PROPERTY_VERSION(), "1"); // backwards compatibility
- context.put(FORMAT_PROPERTY_VERSION(), "1");
-
- return context;
- }
-
- @Override
- public List<String> supportedProperties() {
- List<String> properties = new ArrayList<>();
-
- // kafka
- properties.add(CONNECTOR_TOPIC);
- properties.add(CONNECTOR_PROPERTIES);
- properties.add(CONNECTOR_PROPERTIES + ".#." + CONNECTOR_PROPERTIES_KEY);
- properties.add(CONNECTOR_PROPERTIES + ".#." + CONNECTOR_PROPERTIES_VALUE);
- properties.add(CONNECTOR_STARTUP_MODE);
- properties.add(CONNECTOR_SPECIFIC_OFFSETS + ".#." + CONNECTOR_SPECIFIC_OFFSETS_PARTITION);
- properties.add(CONNECTOR_SPECIFIC_OFFSETS + ".#." + CONNECTOR_SPECIFIC_OFFSETS_OFFSET);
-
- // json format
- properties.add(FORMAT_JSON_SCHEMA);
- properties.add(FORMAT_SCHEMA);
- properties.add(FORMAT_FAIL_ON_MISSING_FIELD);
- properties.add(FORMAT_DERIVE_SCHEMA());
-
- // schema
- properties.add(SCHEMA() + ".#." + SCHEMA_TYPE());
- properties.add(SCHEMA() + ".#." + SCHEMA_NAME());
- properties.add(SCHEMA() + ".#." + SCHEMA_FROM());
-
- // time attributes
- properties.add(SCHEMA() + ".#." + SCHEMA_PROCTIME());
- properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_TYPE());
- properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_FROM());
- properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_CLASS());
- properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_SERIALIZED());
- properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_TYPE());
- properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_CLASS());
- properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_SERIALIZED());
- properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_DELAY());
-
- return properties;
- }
-
- @Override
- public TableSource<Row> create(Map<String, String> properties) {
- final DescriptorProperties params = new DescriptorProperties(true);
- params.putProperties(properties);
-
- // validate
- new SchemaValidator(true).validate(params);
- new KafkaValidator().validate(params);
- new JsonValidator().validate(params);
-
- // build
- final KafkaJsonTableSource.Builder builder = createBuilder();
-
- // topic
- final String topic = params.getString(CONNECTOR_TOPIC);
- builder.forTopic(topic);
-
- // properties
- final Properties props = new Properties();
- final List<Map<String, String>> propsList = params.getFixedIndexedProperties(
- CONNECTOR_PROPERTIES,
- Arrays.asList(CONNECTOR_PROPERTIES_KEY, CONNECTOR_PROPERTIES_VALUE));
- propsList.forEach(kv -> props.put(
- params.getString(kv.get(CONNECTOR_PROPERTIES_KEY)),
- params.getString(kv.get(CONNECTOR_PROPERTIES_VALUE))
- ));
- builder.withKafkaProperties(props);
-
- // startup mode
- params
- .getOptionalString(CONNECTOR_STARTUP_MODE)
- .ifPresent(startupMode -> {
- switch (startupMode) {
-
- case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_EARLIEST:
- builder.fromEarliest();
- break;
-
- case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_LATEST:
- builder.fromLatest();
- break;
-
- case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_GROUP_OFFSETS:
- builder.fromGroupOffsets();
- break;
-
- case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_SPECIFIC_OFFSETS:
- final Map<KafkaTopicPartition, Long> offsetMap = new HashMap<>();
-
- final List<Map<String, String>> offsetList = params.getFixedIndexedProperties(
- CONNECTOR_SPECIFIC_OFFSETS,
- Arrays.asList(CONNECTOR_SPECIFIC_OFFSETS_PARTITION, CONNECTOR_SPECIFIC_OFFSETS_OFFSET));
- offsetList.forEach(kv -> {
- final int partition = params.getInt(kv.get(CONNECTOR_SPECIFIC_OFFSETS_PARTITION));
- final long offset = params.getLong(kv.get(CONNECTOR_SPECIFIC_OFFSETS_OFFSET));
- final KafkaTopicPartition topicPartition = new KafkaTopicPartition(topic, partition);
- offsetMap.put(topicPartition, offset);
- });
- builder.fromSpecificOffsets(offsetMap);
- break;
- }
- });
-
- // missing field
- params.getOptionalBoolean(FORMAT_FAIL_ON_MISSING_FIELD).ifPresent(builder::failOnMissingField);
-
- // json schema
- final TableSchema formatSchema;
- if (params.containsKey(FORMAT_SCHEMA)) {
- final TypeInformation<?> info = params.getType(FORMAT_SCHEMA);
- formatSchema = TableSchema.fromTypeInfo(info);
- } else if (params.containsKey(FORMAT_JSON_SCHEMA)) {
- final TypeInformation<?> info = JsonSchemaConverter.convert(params.getString(FORMAT_JSON_SCHEMA));
- formatSchema = TableSchema.fromTypeInfo(info);
- } else {
- formatSchema = SchemaValidator.deriveFormatFields(params);
- }
- builder.forJsonSchema(formatSchema);
-
- // schema
- final TableSchema schema = params.getTableSchema(SCHEMA());
- builder.withSchema(schema);
-
- // proctime
- SchemaValidator.deriveProctimeAttribute(params).ifPresent(builder::withProctimeAttribute);
-
- // rowtime
- final List<RowtimeAttributeDescriptor> descriptors = SchemaValidator.deriveRowtimeAttributes(params);
- if (descriptors.size() > 1) {
- throw new TableException("More than one rowtime attribute is not supported yet.");
- } else if (descriptors.size() == 1) {
- final RowtimeAttributeDescriptor desc = descriptors.get(0);
- builder.withRowtimeAttribute(desc.getAttributeName(), desc.getTimestampExtractor(), desc.getWatermarkStrategy());
- }
-
- // field mapping
- final Map<String, String> mapping = SchemaValidator.deriveFieldMapping(params, Optional.of(formatSchema));
- builder.withTableToJsonMapping(mapping);
-
- return builder.build();
- }
-
- protected abstract KafkaJsonTableSource.Builder createBuilder();
-
- protected abstract String kafkaVersion();
-}
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java
index 687df589cd0..7853bb702a5 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java
@@ -23,12 +23,17 @@
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.typeutils.RowTypeInfo;
import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
import org.apache.flink.table.sinks.AppendStreamTableSink;
import org.apache.flink.table.util.TableConnectorUtil;
import org.apache.flink.types.Row;
import org.apache.flink.util.Preconditions;
+import java.util.Arrays;
+import java.util.Objects;
+import java.util.Optional;
import java.util.Properties;
/**
@@ -40,27 +45,59 @@
@Internal
public abstract class KafkaTableSink implements AppendStreamTableSink<Row> {
+ // TODO make all attributes final and mandatory once we drop support for format-specific table sinks
+
+ /** The schema of the table. */
+ private final Optional<TableSchema> schema;
+
+ /** The Kafka topic to write to. */
protected final String topic;
+
+ /** Properties for the Kafka producer. */
protected final Properties properties;
- protected SerializationSchema<Row> serializationSchema;
+
+ /** Serialization schema for encoding records to Kafka. */
+ protected Optional<SerializationSchema<Row>> serializationSchema;
+
+ /** Partitioner to select Kafka partition for each item. */
protected final FlinkKafkaPartitioner<Row> partitioner;
+
+ // legacy variables
protected String[] fieldNames;
protected TypeInformation[] fieldTypes;
+ protected KafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema) {
+ this.schema = Optional.of(Preconditions.checkNotNull(schema, "Schema must not be null."));
+ this.topic = Preconditions.checkNotNull(topic, "Topic must not be null.");
+ this.properties = Preconditions.checkNotNull(properties, "Properties must not be null.");
+ this.partitioner = Preconditions.checkNotNull(partitioner, "Partitioner must not be null.");
+ this.serializationSchema = Optional.of(Preconditions.checkNotNull(
+ serializationSchema, "Serialization schema must not be null."));
+ }
+
/**
* Creates KafkaTableSink.
*
* @param topic Kafka topic to write to.
- * @param properties Properties for the Kafka consumer.
+ * @param properties Properties for the Kafka producer.
* @param partitioner Partitioner to select Kafka partition for each item
+ * @deprecated Use table descriptors instead of implementation-specific classes.
*/
+ @Deprecated
public KafkaTableSink(
String topic,
Properties properties,
FlinkKafkaPartitioner<Row> partitioner) {
+ this.schema = Optional.empty();
this.topic = Preconditions.checkNotNull(topic, "topic");
this.properties = Preconditions.checkNotNull(properties, "properties");
this.partitioner = Preconditions.checkNotNull(partitioner, "partitioner");
+ this.serializationSchema = Optional.empty();
}
/**
@@ -72,8 +109,9 @@ public KafkaTableSink(
* @param partitioner Partitioner to select Kafka partition.
* @return The version-specific Kafka producer
*/
- protected abstract FlinkKafkaProducerBase<Row> createKafkaProducer(
- String topic, Properties properties,
+ protected abstract SinkFunction<Row> createKafkaProducer(
+ String topic,
+ Properties properties,
SerializationSchema<Row> serializationSchema,
FlinkKafkaPartitioner<Row> partitioner);
@@ -82,40 +120,57 @@ public KafkaTableSink(
*
* @param rowSchema the schema of the row to serialize.
* @return Instance of serialization schema
+ * @deprecated Use the constructor to pass a serialization schema instead.
*/
- protected abstract SerializationSchema<Row> createSerializationSchema(RowTypeInfo rowSchema);
+ @Deprecated
+ protected SerializationSchema<Row> createSerializationSchema(RowTypeInfo rowSchema) {
+ throw new UnsupportedOperationException("This method only exists for backwards compatibility.");
+ }
/**
* Create a deep copy of this sink.
*
* @return Deep copy of this sink
*/
- protected abstract KafkaTableSink createCopy();
+ @Deprecated
+ protected KafkaTableSink createCopy() {
+ throw new UnsupportedOperationException("This method only exists for backwards compatibility.");
+ }
@Override
public void emitDataStream(DataStream<Row> dataStream) {
- FlinkKafkaProducerBase<Row> kafkaProducer = createKafkaProducer(topic, properties, serializationSchema, partitioner);
- // always enable flush on checkpoint to achieve at-least-once if query runs with checkpointing enabled.
- kafkaProducer.setFlushOnCheckpoint(true);
+ SinkFunction<Row> kafkaProducer = createKafkaProducer(
+ topic,
+ properties,
+ serializationSchema.orElseThrow(() -> new IllegalStateException("No serialization schema defined.")),
+ partitioner);
dataStream.addSink(kafkaProducer).name(TableConnectorUtil.generateRuntimeName(this.getClass(), fieldNames));
}
@Override
public TypeInformation<Row> getOutputType() {
- return new RowTypeInfo(getFieldTypes());
+ return schema
+ .map(TableSchema::toRowType)
+ .orElseGet(() -> new RowTypeInfo(getFieldTypes()));
}
public String[] getFieldNames() {
- return fieldNames;
+ return schema.map(TableSchema::getColumnNames).orElse(fieldNames);
}
@Override
public TypeInformation<?>[] getFieldTypes() {
- return fieldTypes;
+ return schema.map(TableSchema::getTypes).orElse(fieldTypes);
}
@Override
public KafkaTableSink configure(String[] fieldNames, TypeInformation<?>[] fieldTypes) {
+ if (schema.isPresent()) {
+ // a fixed schema is defined so reconfiguration is not supported
+ throw new UnsupportedOperationException("Reconfiguration of this sink is not supported.");
+ }
+
+ // legacy code
KafkaTableSink copy = createCopy();
copy.fieldNames = Preconditions.checkNotNull(fieldNames, "fieldNames");
copy.fieldTypes = Preconditions.checkNotNull(fieldTypes, "fieldTypes");
@@ -123,8 +178,39 @@ public KafkaTableSink configure(String[] fieldNames, TypeInformation<?>[] fieldT
"Number of provided field names and types does not match.");
RowTypeInfo rowSchema = new RowTypeInfo(fieldTypes, fieldNames);
- copy.serializationSchema = createSerializationSchema(rowSchema);
+ copy.serializationSchema = Optional.of(createSerializationSchema(rowSchema));
return copy;
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ KafkaTableSink that = (KafkaTableSink) o;
+ return Objects.equals(schema, that.schema) &&
+ Objects.equals(topic, that.topic) &&
+ Objects.equals(properties, that.properties) &&
+ Objects.equals(serializationSchema, that.serializationSchema) &&
+ Objects.equals(partitioner, that.partitioner) &&
+ Arrays.equals(fieldNames, that.fieldNames) &&
+ Arrays.equals(fieldTypes, that.fieldTypes);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = Objects.hash(
+ schema,
+ topic,
+ properties,
+ serializationSchema,
+ partitioner);
+ result = 31 * result + Arrays.hashCode(fieldNames);
+ result = 31 * result + Arrays.hashCode(fieldTypes);
+ return result;
+ }
}
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java
index 134c483a950..474c22f704f 100644
--- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java
@@ -28,6 +28,8 @@
import org.apache.flink.table.api.TableSchema;
import org.apache.flink.table.api.Types;
import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.descriptors.ConnectorDescriptor;
+import org.apache.flink.table.sources.DefinedFieldMapping;
import org.apache.flink.table.sources.DefinedProctimeAttribute;
import org.apache.flink.table.sources.DefinedRowtimeAttributes;
import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
@@ -43,6 +45,7 @@
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.Properties;
import scala.Option;
@@ -54,26 +57,37 @@
* override {@link #createKafkaConsumer(String, Properties, DeserializationSchema)}}.
*/
@Internal
-public abstract class KafkaTableSource
- implements StreamTableSource<Row>, DefinedProctimeAttribute, DefinedRowtimeAttributes {
+public abstract class KafkaTableSource implements
+ StreamTableSource<Row>,
+ DefinedProctimeAttribute,
+ DefinedRowtimeAttributes,
+ DefinedFieldMapping {
+
+ // common table source attributes
+ // TODO make all attributes final once we drop support for format-specific table sources
/** The schema of the table. */
private final TableSchema schema;
+ /** Field name of the processing time attribute, null if no processing time field is defined. */
+ private Optional<String> proctimeAttribute;
+
+ /** Descriptor for a rowtime attribute. */
+ private List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors;
+
+ /** Mapping for the fields of the table schema to fields of the physical returned type. */
+ private Optional<Map<String, String>> fieldMapping;
+
+ // Kafka-specific attributes
+
/** The Kafka topic to consume. */
private final String topic;
/** Properties for the Kafka consumer. */
private final Properties properties;
- /** Type information describing the result type. */
- private TypeInformation<Row> returnType;
-
- /** Field name of the processing time attribute, null if no processing time field is defined. */
- private String proctimeAttribute;
-
- /** Descriptor for a rowtime attribute. */
- private List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors;
+ /** Deserialization schema for decoding records from Kafka. */
+ private final DeserializationSchema<Row> deserializationSchema;
/** The startup mode for the contained consumer (default is {@link StartupMode#GROUP_OFFSETS}). */
private StartupMode startupMode;
@@ -84,21 +98,63 @@
/**
* Creates a generic Kafka {@link StreamTableSource}.
*
- * @param topic Kafka topic to consume.
- * @param properties Properties for the Kafka consumer.
- * @param schema Schema of the produced table.
- * @param returnType Type information of the produced physical DataStream.
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*/
protected KafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Optional<Map<String, String>> fieldMapping,
String topic,
Properties properties,
- TableSchema schema,
- TypeInformation<Row> returnType) {
-
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets) {
+ this.schema = Preconditions.checkNotNull(schema, "Schema must not be null.");
+ this.proctimeAttribute = validateProctimeAttribute(proctimeAttribute);
+ this.rowtimeAttributeDescriptors = validateRowtimeAttributeDescriptors(rowtimeAttributeDescriptors);
+ this.fieldMapping = fieldMapping;
this.topic = Preconditions.checkNotNull(topic, "Topic must not be null.");
this.properties = Preconditions.checkNotNull(properties, "Properties must not be null.");
- this.schema = Preconditions.checkNotNull(schema, "Schema must not be null.");
- this.returnType = Preconditions.checkNotNull(returnType, "Type information must not be null.");
+ this.deserializationSchema = Preconditions.checkNotNull(
+ deserializationSchema, "Deserialization schema must not be null.");
+ this.startupMode = Preconditions.checkNotNull(startupMode, "Startup mode must not be null.");
+ this.specificStartupOffsets = Preconditions.checkNotNull(
+ specificStartupOffsets, "Specific offsets must not be null.");
+ }
+
+ /**
+ * Creates a generic Kafka {@link StreamTableSource}.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ */
+ protected KafkaTableSource(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema) {
+ this(
+ schema,
+ Optional.empty(),
+ Collections.emptyList(),
+ Optional.empty(),
+ topic, properties,
+ deserializationSchema,
+ StartupMode.GROUP_OFFSETS,
+ Collections.emptyMap());
}
/**
@@ -116,7 +172,7 @@ protected KafkaTableSource(
@Override
public TypeInformation<Row> getReturnType() {
- return returnType;
+ return deserializationSchema.getProducedType();
}
@Override
@@ -126,7 +182,7 @@ public TableSchema getTableSchema() {
@Override
public String getProctimeAttribute() {
- return proctimeAttribute;
+ return proctimeAttribute.orElse(null);
}
@Override
@@ -134,34 +190,68 @@ public String getProctimeAttribute() {
return rowtimeAttributeDescriptors;
}
+ @Override
+ public Map<String, String> getFieldMapping() {
+ return fieldMapping.orElse(null);
+ }
+
@Override
public String explainSource() {
return TableConnectorUtil.generateRuntimeName(this.getClass(), schema.getColumnNames());
}
+ /**
+ * Returns the properties for the Kafka consumer.
+ *
+ * @return properties for the Kafka consumer.
+ */
+ public Properties getProperties() {
+ return properties;
+ }
+
+ /**
+ * Returns the deserialization schema.
+ *
+ * @return The deserialization schema
+ */
+ public DeserializationSchema<Row> getDeserializationSchema(){
+ return deserializationSchema;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
- if (!(o instanceof KafkaTableSource)) {
+ // TODO force classes to be equal once we drop support for format-specific table sources
+ // if (o == null || getClass() != o.getClass()) {
+ if (o == null || !(o instanceof KafkaTableSource)) {
return false;
}
- KafkaTableSource that = (KafkaTableSource) o;
+ final KafkaTableSource that = (KafkaTableSource) o;
return Objects.equals(schema, that.schema) &&
- Objects.equals(topic, that.topic) &&
- Objects.equals(properties, that.properties) &&
- Objects.equals(returnType, that.returnType) &&
Objects.equals(proctimeAttribute, that.proctimeAttribute) &&
Objects.equals(rowtimeAttributeDescriptors, that.rowtimeAttributeDescriptors) &&
+ Objects.equals(fieldMapping, that.fieldMapping) &&
+ Objects.equals(topic, that.topic) &&
+ Objects.equals(properties, that.properties) &&
+ Objects.equals(deserializationSchema, that.deserializationSchema) &&
startupMode == that.startupMode &&
Objects.equals(specificStartupOffsets, that.specificStartupOffsets);
}
@Override
public int hashCode() {
- return Objects.hash(schema, topic, properties, returnType,
- proctimeAttribute, rowtimeAttributeDescriptors, startupMode, specificStartupOffsets);
+ return Objects.hash(
+ schema,
+ proctimeAttribute,
+ rowtimeAttributeDescriptors,
+ fieldMapping,
+ topic,
+ properties,
+ deserializationSchema,
+ startupMode,
+ specificStartupOffsets);
}
/**
@@ -195,32 +285,33 @@ public int hashCode() {
return kafkaConsumer;
}
- //////// SETTERS FOR OPTIONAL PARAMETERS
+ //////// VALIDATION FOR PARAMETERS
/**
- * Declares a field of the schema to be the processing time attribute.
+ * Validates a field of the schema to be the processing time attribute.
*
* @param proctimeAttribute The name of the field that becomes the processing time field.
*/
- protected void setProctimeAttribute(String proctimeAttribute) {
- if (proctimeAttribute != null) {
+ private Optional<String> validateProctimeAttribute(Optional<String> proctimeAttribute) {
+ return proctimeAttribute.map((attribute) -> {
// validate that field exists and is of correct type
- Option<TypeInformation<?>> tpe = schema.getType(proctimeAttribute);
+ Option<TypeInformation<?>> tpe = schema.getType(attribute);
if (tpe.isEmpty()) {
- throw new ValidationException("Processing time attribute '" + proctimeAttribute + "' is not present in TableSchema.");
+ throw new ValidationException("Processing time attribute '" + attribute + "' is not present in TableSchema.");
} else if (tpe.get() != Types.SQL_TIMESTAMP()) {
- throw new ValidationException("Processing time attribute '" + proctimeAttribute + "' is not of type SQL_TIMESTAMP.");
+ throw new ValidationException("Processing time attribute '" + attribute + "' is not of type SQL_TIMESTAMP.");
}
- }
- this.proctimeAttribute = proctimeAttribute;
+ return attribute;
+ });
}
/**
- * Declares a list of fields to be rowtime attributes.
+ * Validates a list of fields to be rowtime attributes.
*
* @param rowtimeAttributeDescriptors The descriptors of the rowtime attributes.
*/
- protected void setRowtimeAttributeDescriptors(List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors) {
+ private List<RowtimeAttributeDescriptor> validateRowtimeAttributeDescriptors(List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors) {
+ Preconditions.checkNotNull(rowtimeAttributeDescriptors, "List of rowtime attributes must not be null.");
// validate that all declared fields exist and are of correct type
for (RowtimeAttributeDescriptor desc : rowtimeAttributeDescriptors) {
String rowtimeAttribute = desc.getAttributeName();
@@ -231,25 +322,64 @@ protected void setRowtimeAttributeDescriptors(List<RowtimeAttributeDescriptor> r
throw new ValidationException("Rowtime attribute '" + rowtimeAttribute + "' is not of type SQL_TIMESTAMP.");
}
}
- this.rowtimeAttributeDescriptors = rowtimeAttributeDescriptors;
+ return rowtimeAttributeDescriptors;
+ }
+
+ //////// SETTERS FOR OPTIONAL PARAMETERS
+
+ /**
+ * Declares a field of the schema to be the processing time attribute.
+ *
+ * @param proctimeAttribute The name of the field that becomes the processing time field.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
+ */
+ @Deprecated
+ protected void setProctimeAttribute(String proctimeAttribute) {
+ this.proctimeAttribute = validateProctimeAttribute(Optional.ofNullable(proctimeAttribute));
+ }
+
+ /**
+ * Declares a list of fields to be rowtime attributes.
+ *
+ * @param rowtimeAttributeDescriptors The descriptors of the rowtime attributes.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
+ */
+ @Deprecated
+ protected void setRowtimeAttributeDescriptors(List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors) {
+ this.rowtimeAttributeDescriptors = validateRowtimeAttributeDescriptors(rowtimeAttributeDescriptors);
}
/**
* Sets the startup mode of the TableSource.
*
* @param startupMode The startup mode.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected void setStartupMode(StartupMode startupMode) {
- this.startupMode = startupMode;
+ this.startupMode = Preconditions.checkNotNull(startupMode);
}
/**
* Sets the startup offsets of the TableSource; only relevant when the startup mode is {@link StartupMode#SPECIFIC_OFFSETS}.
*
* @param specificStartupOffsets The startup offsets for different partitions.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected void setSpecificStartupOffsets(Map<KafkaTopicPartition, Long> specificStartupOffsets) {
- this.specificStartupOffsets = specificStartupOffsets;
+ this.specificStartupOffsets = Preconditions.checkNotNull(specificStartupOffsets);
+ }
+
+ /**
+ * Mapping for the fields of the table schema to fields of the physical returned type.
+ *
+ * @param fieldMapping The mapping from table schema fields to format schema fields.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
+ */
+ @Deprecated
+ protected void setFieldMapping(Map<String, String> fieldMapping) {
+ this.fieldMapping = Optional.ofNullable(fieldMapping);
}
//////// ABSTRACT METHODS FOR SUBCLASSES
@@ -267,20 +397,18 @@ protected void setSpecificStartupOffsets(Map<KafkaTopicPartition, Long> specific
Properties properties,
DeserializationSchema<Row> deserializationSchema);
- /**
- * Returns the deserialization schema.
- *
- * @return The deserialization schema
- */
- protected abstract DeserializationSchema<Row> getDeserializationSchema();
-
/**
* Abstract builder for a {@link KafkaTableSource} to be extended by builders of subclasses of
* KafkaTableSource.
*
* @param <T> Type of the KafkaTableSource produced by the builder.
* @param <B> Type of the KafkaTableSource.Builder subclass.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
protected abstract static class Builder<T extends KafkaTableSource, B extends KafkaTableSource.Builder> {
private String topic;
@@ -299,13 +427,14 @@ protected void setSpecificStartupOffsets(Map<KafkaTopicPartition, Long> specific
/** Specific startup offsets; only relevant when startup mode is {@link StartupMode#SPECIFIC_OFFSETS}. */
private Map<KafkaTopicPartition, Long> specificStartupOffsets = null;
-
/**
* Sets the topic from which the table is read.
*
* @param topic The topic from which the table is read.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B forTopic(String topic) {
Preconditions.checkNotNull(topic, "Topic must not be null.");
Preconditions.checkArgument(this.topic == null, "Topic has already been set.");
@@ -318,7 +447,9 @@ public B forTopic(String topic) {
*
* @param props The configuration properties for the Kafka consumer.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withKafkaProperties(Properties props) {
Preconditions.checkNotNull(props, "Properties must not be null.");
Preconditions.checkArgument(this.kafkaProps == null, "Properties have already been set.");
@@ -331,7 +462,9 @@ public B withKafkaProperties(Properties props) {
*
* @param schema The schema of the produced table.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withSchema(TableSchema schema) {
Preconditions.checkNotNull(schema, "Schema must not be null.");
Preconditions.checkArgument(this.schema == null, "Schema has already been set.");
@@ -345,7 +478,9 @@ public B withSchema(TableSchema schema) {
*
* @param proctimeAttribute The name of the processing time attribute in the table schema.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withProctimeAttribute(String proctimeAttribute) {
Preconditions.checkNotNull(proctimeAttribute, "Proctime attribute must not be null.");
Preconditions.checkArgument(!proctimeAttribute.isEmpty(), "Proctime attribute must not be empty.");
@@ -362,7 +497,9 @@ public B withProctimeAttribute(String proctimeAttribute) {
* @param timestampExtractor The {@link TimestampExtractor} to extract the rowtime attribute from the physical type.
* @param watermarkStrategy The {@link WatermarkStrategy} to generate watermarks for the rowtime attribute.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withRowtimeAttribute(
String rowtimeAttribute,
TimestampExtractor timestampExtractor,
@@ -389,7 +526,9 @@ public B withRowtimeAttribute(
* @param rowtimeAttribute The name of the rowtime attribute in the table schema.
* @param watermarkStrategy The {@link WatermarkStrategy} to generate watermarks for the rowtime attribute.
* @return The builder.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B withKafkaTimestampAsRowtimeAttribute(
String rowtimeAttribute,
WatermarkStrategy watermarkStrategy) {
@@ -412,7 +551,9 @@ public B withKafkaTimestampAsRowtimeAttribute(
* Configures the TableSource to start reading from the earliest offset for all partitions.
*
* @see FlinkKafkaConsumerBase#setStartFromEarliest()
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B fromEarliest() {
this.startupMode = StartupMode.EARLIEST;
this.specificStartupOffsets = null;
@@ -423,7 +564,9 @@ public B fromEarliest() {
* Configures the TableSource to start reading from the latest offset for all partitions.
*
* @see FlinkKafkaConsumerBase#setStartFromLatest()
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B fromLatest() {
this.startupMode = StartupMode.LATEST;
this.specificStartupOffsets = null;
@@ -434,7 +577,9 @@ public B fromLatest() {
* Configures the TableSource to start reading from any committed group offsets found in Zookeeper / Kafka brokers.
*
* @see FlinkKafkaConsumerBase#setStartFromGroupOffsets()
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B fromGroupOffsets() {
this.startupMode = StartupMode.GROUP_OFFSETS;
this.specificStartupOffsets = null;
@@ -446,7 +591,9 @@ public B fromGroupOffsets() {
*
* @param specificStartupOffsets the specified offsets for partitions
* @see FlinkKafkaConsumerBase#setStartFromSpecificOffsets(Map)
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
public B fromSpecificOffsets(Map<KafkaTopicPartition, Long> specificStartupOffsets) {
this.startupMode = StartupMode.SPECIFIC_OFFSETS;
this.specificStartupOffsets = Preconditions.checkNotNull(specificStartupOffsets);
@@ -457,7 +604,9 @@ public B fromSpecificOffsets(Map<KafkaTopicPartition, Long> specificStartupOffse
* Returns the configured topic.
*
* @return the configured topic.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected String getTopic() {
return this.topic;
}
@@ -466,7 +615,9 @@ protected String getTopic() {
* Returns the configured Kafka properties.
*
* @return the configured Kafka properties.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected Properties getKafkaProps() {
return this.kafkaProps;
}
@@ -475,7 +626,9 @@ protected Properties getKafkaProps() {
* Returns the configured table schema.
*
* @return the configured table schema.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected TableSchema getTableSchema() {
return this.schema;
}
@@ -484,14 +637,18 @@ protected TableSchema getTableSchema() {
* True if the KafkaSource supports Kafka timestamps, false otherwise.
*
* @return True if the KafkaSource supports Kafka timestamps, false otherwise.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected abstract boolean supportsKafkaTimestamps();
/**
* Configures a TableSource with optional parameters.
*
* @param tableSource The TableSource to configure.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected void configureTableSource(T tableSource) {
// configure processing time attributes
tableSource.setProctimeAttribute(proctimeAttribute);
@@ -516,13 +673,20 @@ protected void configureTableSource(T tableSource) {
/**
* Returns the builder.
* @return the builder.
+ * @deprecated Use the {@link org.apache.flink.table.descriptors.Kafka} descriptor together
+ * with descriptors for schema and format instead. Descriptors allow for
+ * implementation-agnostic definition of tables. See also
+ * {@link org.apache.flink.table.api.TableEnvironment#connect(ConnectorDescriptor)}.
*/
+ @Deprecated
protected abstract B builder();
/**
* Builds the configured {@link KafkaTableSource}.
* @return The configured {@link KafkaTableSource}.
+ * @deprecated Use table descriptors instead of implementation-specific builders.
*/
+ @Deprecated
protected abstract KafkaTableSource build();
}
}
diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSourceSinkFactoryBase.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSourceSinkFactoryBase.java
new file mode 100644
index 00000000000..27b2e67ce0b
--- /dev/null
+++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSourceSinkFactoryBase.java
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.streaming.connectors.kafka.config.StartupMode;
+import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner;
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.descriptors.DescriptorProperties;
+import org.apache.flink.table.descriptors.KafkaValidator;
+import org.apache.flink.table.descriptors.SchemaValidator;
+import org.apache.flink.table.factories.DeserializationSchemaFactory;
+import org.apache.flink.table.factories.SerializationSchemaFactory;
+import org.apache.flink.table.factories.StreamTableSinkFactory;
+import org.apache.flink.table.factories.StreamTableSourceFactory;
+import org.apache.flink.table.factories.TableFactoryService;
+import org.apache.flink.table.sinks.StreamTableSink;
+import org.apache.flink.table.sources.RowtimeAttributeDescriptor;
+import org.apache.flink.table.sources.StreamTableSource;
+import org.apache.flink.types.Row;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Properties;
+
+import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_PROPERTY_VERSION;
+import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_TYPE;
+import static org.apache.flink.table.descriptors.ConnectorDescriptorValidator.CONNECTOR_VERSION;
+import static org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES_KEY;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_PROPERTIES_VALUE;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS_OFFSET;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_SPECIFIC_OFFSETS_PARTITION;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_STARTUP_MODE;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_TOPIC;
+import static org.apache.flink.table.descriptors.KafkaValidator.CONNECTOR_TYPE_VALUE_KAFKA;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_CLASS;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_FROM;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_SERIALIZED;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_TIMESTAMPS_TYPE;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_CLASS;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_DELAY;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_SERIALIZED;
+import static org.apache.flink.table.descriptors.RowtimeValidator.ROWTIME_WATERMARKS_TYPE;
+import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA;
+import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_FROM;
+import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_NAME;
+import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_PROCTIME;
+import static org.apache.flink.table.descriptors.SchemaValidator.SCHEMA_TYPE;
+import static org.apache.flink.table.descriptors.StreamTableDescriptorValidator.UPDATE_MODE;
+import static org.apache.flink.table.descriptors.StreamTableDescriptorValidator.UPDATE_MODE_VALUE_APPEND;
+
+/**
+ * Factory for creating configured instances of {@link KafkaTableSource}.
+ */
+public abstract class KafkaTableSourceSinkFactoryBase implements
+ StreamTableSourceFactory<Row>,
+ StreamTableSinkFactory<Row> {
+
+ @Override
+ public Map<String, String> requiredContext() {
+ Map<String, String> context = new HashMap<>();
+ context.put(UPDATE_MODE(), UPDATE_MODE_VALUE_APPEND()); // append mode
+ context.put(CONNECTOR_TYPE(), CONNECTOR_TYPE_VALUE_KAFKA); // kafka
+ context.put(CONNECTOR_VERSION(), kafkaVersion()); // version
+ context.put(CONNECTOR_PROPERTY_VERSION(), "1"); // backwards compatibility
+ return context;
+ }
+
+ @Override
+ public List<String> supportedProperties() {
+ List<String> properties = new ArrayList<>();
+
+ // kafka
+ properties.add(CONNECTOR_TOPIC);
+ properties.add(CONNECTOR_PROPERTIES);
+ properties.add(CONNECTOR_PROPERTIES + ".#." + CONNECTOR_PROPERTIES_KEY);
+ properties.add(CONNECTOR_PROPERTIES + ".#." + CONNECTOR_PROPERTIES_VALUE);
+ properties.add(CONNECTOR_STARTUP_MODE);
+ properties.add(CONNECTOR_SPECIFIC_OFFSETS + ".#." + CONNECTOR_SPECIFIC_OFFSETS_PARTITION);
+ properties.add(CONNECTOR_SPECIFIC_OFFSETS + ".#." + CONNECTOR_SPECIFIC_OFFSETS_OFFSET);
+
+ // schema
+ properties.add(SCHEMA() + ".#." + SCHEMA_TYPE());
+ properties.add(SCHEMA() + ".#." + SCHEMA_NAME());
+ properties.add(SCHEMA() + ".#." + SCHEMA_FROM());
+
+ // time attributes
+ properties.add(SCHEMA() + ".#." + SCHEMA_PROCTIME());
+ properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_TYPE());
+ properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_FROM());
+ properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_CLASS());
+ properties.add(SCHEMA() + ".#." + ROWTIME_TIMESTAMPS_SERIALIZED());
+ properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_TYPE());
+ properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_CLASS());
+ properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_SERIALIZED());
+ properties.add(SCHEMA() + ".#." + ROWTIME_WATERMARKS_DELAY());
+
+ // format wildcard
+ properties.add(FORMAT() + ".*");
+
+ return properties;
+ }
+
+ @Override
+ public StreamTableSource<Row> createStreamTableSource(Map<String, String> properties) {
+ final DescriptorProperties descriptorProperties = getValidatedProperties(properties);
+
+ final String topic = descriptorProperties.getString(CONNECTOR_TOPIC);
+ final DeserializationSchema<Row> deserializationSchema = getDeserializationSchema(properties);
+ final StartupOptions startupOptions = getStartupOptions(descriptorProperties, topic);
+
+ return createKafkaTableSource(
+ descriptorProperties.getTableSchema(SCHEMA()),
+ SchemaValidator.deriveProctimeAttribute(descriptorProperties),
+ SchemaValidator.deriveRowtimeAttributes(descriptorProperties),
+ SchemaValidator.deriveFieldMapping(
+ descriptorProperties,
+ Optional.of(deserializationSchema.getProducedType())),
+ topic,
+ getKafkaProperties(descriptorProperties),
+ deserializationSchema,
+ startupOptions.startupMode,
+ startupOptions.specificOffsets);
+ }
+
+ @Override
+ public StreamTableSink<Row> createStreamTableSink(Map<String, String> properties) {
+ final DescriptorProperties descriptorProperties = getValidatedProperties(properties);
+
+ final TableSchema schema = descriptorProperties.getTableSchema(SCHEMA());
+ final String topic = descriptorProperties.getString(CONNECTOR_TOPIC);
+ final Optional<String> proctime = SchemaValidator.deriveProctimeAttribute(descriptorProperties);
+ final List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors =
+ SchemaValidator.deriveRowtimeAttributes(descriptorProperties);
+
+ // see also FLINK-9870
+ if (proctime.isPresent() || !rowtimeAttributeDescriptors.isEmpty() ||
+ checkForCustomFieldMapping(descriptorProperties, schema)) {
+ throw new TableException("Time attributes and custom field mappings are not supported yet.");
+ }
+
+ return createKafkaTableSink(
+ schema,
+ topic,
+ getKafkaProperties(descriptorProperties),
+ getFlinkKafkaPartitioner(),
+ getSerializationSchema(properties));
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // For version-specific factories
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Returns the Kafka version.
+ */
+ protected abstract String kafkaVersion();
+
+ /**
+ * True if the Kafka source supports Kafka timestamps, false otherwise.
+ *
+ * @return True if the Kafka source supports Kafka timestamps, false otherwise.
+ */
+ protected abstract boolean supportsKafkaTimestamps();
+
+ /**
+ * Constructs the version-specific Kafka table source.
+ *
+ * @param schema Schema of the produced table.
+ * @param proctimeAttribute Field name of the processing time attribute.
+ * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+ * @param fieldMapping Mapping for the fields of the table schema to
+ * fields of the physical returned type.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param deserializationSchema Deserialization schema for decoding records from Kafka.
+ * @param startupMode Startup mode for the contained consumer.
+ * @param specificStartupOffsets Specific startup offsets; only relevant when startup
+ * mode is {@link StartupMode#SPECIFIC_OFFSETS}.
+ */
+ protected abstract KafkaTableSource createKafkaTableSource(
+ TableSchema schema,
+ Optional<String> proctimeAttribute,
+ List<RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
+ Map<String, String> fieldMapping,
+ String topic,
+ Properties properties,
+ DeserializationSchema<Row> deserializationSchema,
+ StartupMode startupMode,
+ Map<KafkaTopicPartition, Long> specificStartupOffsets);
+
+ /**
+ * Constructs the version-specific Kafka table sink.
+ *
+ * @param schema Schema of the produced table.
+ * @param topic Kafka topic to consume.
+ * @param properties Properties for the Kafka consumer.
+ * @param partitioner Partitioner to select Kafka partition for each item.
+ */
+ protected abstract KafkaTableSink createKafkaTableSink(
+ TableSchema schema,
+ String topic,
+ Properties properties,
+ FlinkKafkaPartitioner<Row> partitioner,
+ SerializationSchema<Row> serializationSchema);
+
+ // --------------------------------------------------------------------------------------------
+ // Helper methods
+ // --------------------------------------------------------------------------------------------
+
+ private DescriptorProperties getValidatedProperties(Map<String, String> properties) {
+ final DescriptorProperties descriptorProperties = new DescriptorProperties(true);
+ descriptorProperties.putProperties(properties);
+
+ // allow Kafka timestamps to be used, watermarks can not be received from source
+ new SchemaValidator(true, supportsKafkaTimestamps(), false).validate(descriptorProperties);
+ new KafkaValidator().validate(descriptorProperties);
+
+ return descriptorProperties;
+ }
+
+ private DeserializationSchema<Row> getDeserializationSchema(Map<String, String> properties) {
+ @SuppressWarnings("unchecked")
+ final DeserializationSchemaFactory<Row> formatFactory = TableFactoryService.find(
+ DeserializationSchemaFactory.class,
+ properties,
+ this.getClass().getClassLoader());
+ return formatFactory.createDeserializationSchema(properties);
+ }
+
+ private SerializationSchema<Row> getSerializationSchema(Map<String, String> properties) {
+ @SuppressWarnings("unchecked")
+ final SerializationSchemaFactory<Row> formatFactory = TableFactoryService.find(
+ SerializationSchemaFactory.class,
+ properties,
+ this.getClass().getClassLoader());
+ return formatFactory.createSerializationSchema(properties);
+ }
+
+ private Properties getKafkaProperties(DescriptorProperties descriptorProperties) {
+ final Properties kafkaProperties = new Properties();
+ final List<Map<String, String>> propsList = descriptorProperties.getFixedIndexedProperties(
+ CONNECTOR_PROPERTIES,
+ Arrays.asList(CONNECTOR_PROPERTIES_KEY, CONNECTOR_PROPERTIES_VALUE));
+ propsList.forEach(kv -> kafkaProperties.put(
+ descriptorProperties.getString(kv.get(CONNECTOR_PROPERTIES_KEY)),
+ descriptorProperties.getString(kv.get(CONNECTOR_PROPERTIES_VALUE))
+ ));
+ return kafkaProperties;
+ }
+
+ private StartupOptions getStartupOptions(
+ DescriptorProperties descriptorProperties,
+ String topic) {
+ final Map<KafkaTopicPartition, Long> specificOffsets = new HashMap<>();
+ final StartupMode startupMode = descriptorProperties
+ .getOptionalString(CONNECTOR_STARTUP_MODE)
+ .map(modeString -> {
+ switch (modeString) {
+ case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_EARLIEST:
+ return StartupMode.EARLIEST;
+
+ case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_LATEST:
+ return StartupMode.LATEST;
+
+ case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_GROUP_OFFSETS:
+ return StartupMode.GROUP_OFFSETS;
+
+ case KafkaValidator.CONNECTOR_STARTUP_MODE_VALUE_SPECIFIC_OFFSETS:
+ final List<Map<String, String>> offsetList = descriptorProperties.getFixedIndexedProperties(
+ CONNECTOR_SPECIFIC_OFFSETS,
+ Arrays.asList(CONNECTOR_SPECIFIC_OFFSETS_PARTITION, CONNECTOR_SPECIFIC_OFFSETS_OFFSET));
+ offsetList.forEach(kv -> {
+ final int partition = descriptorProperties.getInt(kv.get(CONNECTOR_SPECIFIC_OFFSETS_PARTITION));
+ final long offset = descriptorProperties.getLong(kv.get(CONNECTOR_SPECIFIC_OFFSETS_OFFSET));
+ final KafkaTopicPartition topicPartition = new KafkaTopicPartition(topic, partition);
+ specificOffsets.put(topicPartition, offset);
+ });
+ return StartupMode.SPECIFIC_OFFSETS;
+ default:
+ throw new TableException("Unsupported startup mode. Validator should have checked that.");
+ }
+ }).orElse(StartupMode.GROUP_OFFSETS);
+ final StartupOptions options = new StartupOptions();
+ options.startupMode = startupMode;
+ options.specificOffsets = specificOffsets;
+ return options;
+ }
+
+ private FlinkKafkaPartitioner<Row> getFlinkKafkaPartitioner() {
+ // we don't support custom partitioner so far
+ return new FlinkFixedPartitioner<>();
+ }
+
+ private boolean checkForCustomFieldMapping(DescriptorProperties descriptorProperties, TableSchema schema) {
+ final Map<String, String> fieldMapping = SchemaValidator.deriveFieldMapping(
+ descriptorProperties,
+ Optional.of(schema.toRowType())); // until FLINK-9870 is fixed we assume that the table schema is the output type
+ return fieldMapping.size() != schema.getColumnNames().length ||
+ !fieldMapping.entrySet().stream().allMatch(mapping -> mapping.getKey().equals(mapping.getValue()));
+ }
+
+ private static class StartupOptions {
+ private StartupMode startupMode;
(This diff was longer than 20,000 lines, and has been truncated...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services