You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by pn...@apache.org on 2019/08/01 08:31:12 UTC
[flink] branch master updated (6c1a26d -> 57f7958)
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 6c1a26d [FLINK-13511][kafka][build] Add test jaxb dependency
new a170d9e [hotfix][docs] Reference file systems documentation from batch/connectors docs
new 4f8c573 [FLINK-12998][docs] Document optional file systems libs to use plugins loading mechanism
new 57f7958 [FLINK-12998][docs] Extend debugging_classloading documentation with plugins loading case
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/dev/batch/connectors.md | 16 ++-----
docs/dev/batch/connectors.zh.md | 16 ++-----
docs/monitoring/debugging_classloading.md | 55 +++++++++++----------
docs/monitoring/debugging_classloading.zh.md | 55 +++++++++++----------
docs/ops/filesystems/azure.md | 11 +++--
docs/ops/filesystems/azure.zh.md | 11 +++--
docs/ops/filesystems/index.md | 72 +++++++++++++++++++---------
docs/ops/filesystems/index.zh.md | 72 +++++++++++++++++++---------
docs/ops/filesystems/oss.md | 12 +++--
docs/ops/filesystems/oss.zh.md | 12 +++--
docs/ops/filesystems/s3.md | 25 +++++-----
docs/ops/filesystems/s3.zh.md | 25 +++++-----
12 files changed, 216 insertions(+), 166 deletions(-)
[flink] 02/03: [FLINK-12998][docs] Document optional file systems
libs to use plugins loading mechanism
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 4f8c573d19999a2f422e4fe66e357b41b482b336
Author: Aleksey Pak <al...@ververica.com>
AuthorDate: Thu Jul 18 14:22:47 2019 +0200
[FLINK-12998][docs] Document optional file systems libs to use plugins loading mechanism
---
docs/ops/filesystems/azure.md | 11 +++---
docs/ops/filesystems/azure.zh.md | 11 +++---
docs/ops/filesystems/index.md | 72 +++++++++++++++++++++++++++-------------
docs/ops/filesystems/index.zh.md | 72 +++++++++++++++++++++++++++-------------
docs/ops/filesystems/oss.md | 12 ++++---
docs/ops/filesystems/oss.zh.md | 12 ++++---
docs/ops/filesystems/s3.md | 25 +++++++-------
docs/ops/filesystems/s3.zh.md | 25 +++++++-------
8 files changed, 150 insertions(+), 90 deletions(-)
diff --git a/docs/ops/filesystems/azure.md b/docs/ops/filesystems/azure.md
index 36720c8..d721be5 100644
--- a/docs/ops/filesystems/azure.md
+++ b/docs/ops/filesystems/azure.md
@@ -38,7 +38,7 @@ wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path
wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>
{% endhighlight %}
-Below shows how to use Azure Blob Storage with Flink:
+See below for how to use Azure Blob Storage in a Flink job:
{% highlight java %}
// Read from Azure Blob storage
@@ -51,12 +51,13 @@ stream.writeAsText("wasb://<your-container>@$<your-azure-account>.blob.core.wind
env.setStateBackend(new FsStateBackend("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"));
{% endhighlight %}
-### Shaded Hadoop Azure Blob Storage file system
+### Shaded Hadoop Azure Blob Storage file system
-To use `flink-azure-fs-hadoop,` copy the respective JAR file from the opt directory to the lib directory of your Flink distribution before starting Flink, e.g.
+To use `flink-azure-fs-hadoop,` copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-azure-fs-hadoop-{{ site.version }}.jar ./lib/
+mkdir ./plugins/azure-fs-hadoop
+cp ./opt/flink-azure-fs-hadoop-{{ site.version }}.jar ./plugins/azure-fs-hadoop/
{% endhighlight %}
`flink-azure-fs-hadoop` registers default FileSystem wrappers for URIs with the *wasb://* and *wasbs://* (SSL encrypted access) scheme.
@@ -74,4 +75,4 @@ There are some required configurations that must be added to `flink-conf.yaml`:
fs.azure.account.key.youraccount.blob.core.windows.net: Azure Blob Storage access key
{% endhighlight %}
-{% top %}
+{% top %}
diff --git a/docs/ops/filesystems/azure.zh.md b/docs/ops/filesystems/azure.zh.md
index 36720c8..d721be5 100644
--- a/docs/ops/filesystems/azure.zh.md
+++ b/docs/ops/filesystems/azure.zh.md
@@ -38,7 +38,7 @@ wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path
wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>
{% endhighlight %}
-Below shows how to use Azure Blob Storage with Flink:
+See below for how to use Azure Blob Storage in a Flink job:
{% highlight java %}
// Read from Azure Blob storage
@@ -51,12 +51,13 @@ stream.writeAsText("wasb://<your-container>@$<your-azure-account>.blob.core.wind
env.setStateBackend(new FsStateBackend("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"));
{% endhighlight %}
-### Shaded Hadoop Azure Blob Storage file system
+### Shaded Hadoop Azure Blob Storage file system
-To use `flink-azure-fs-hadoop,` copy the respective JAR file from the opt directory to the lib directory of your Flink distribution before starting Flink, e.g.
+To use `flink-azure-fs-hadoop,` copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-azure-fs-hadoop-{{ site.version }}.jar ./lib/
+mkdir ./plugins/azure-fs-hadoop
+cp ./opt/flink-azure-fs-hadoop-{{ site.version }}.jar ./plugins/azure-fs-hadoop/
{% endhighlight %}
`flink-azure-fs-hadoop` registers default FileSystem wrappers for URIs with the *wasb://* and *wasbs://* (SSL encrypted access) scheme.
@@ -74,4 +75,4 @@ There are some required configurations that must be added to `flink-conf.yaml`:
fs.azure.account.key.youraccount.blob.core.windows.net: Azure Blob Storage access key
{% endhighlight %}
-{% top %}
+{% top %}
diff --git a/docs/ops/filesystems/index.md b/docs/ops/filesystems/index.md
index eb4087d..88f5133 100644
--- a/docs/ops/filesystems/index.md
+++ b/docs/ops/filesystems/index.md
@@ -25,7 +25,7 @@ under the License.
-->
Apache Flink uses file systems to consume and persistently store data, both for the results of applications and for fault tolerance and recovery.
-These are some of most of the popular file systems, including *local*, *hadoop-compatible*, *S3*, *MapR FS*, *OpenStack Swift FS*, *Aliyun OSS* and *Azure Blob Storage*.
+These are some of most of the popular file systems, including *local*, *hadoop-compatible*, *Amazon S3*, *MapR FS*, *OpenStack Swift FS*, *Aliyun OSS* and *Azure Blob Storage*.
The file system used for a particular file is determined by its URI scheme.
For example, `file:///home/user/text.txt` refers to a file in the local file system, while `hdfs://namenode:50010/data/user/text.txt` is a file in a specific HDFS cluster.
@@ -35,32 +35,54 @@ File system instances are instantiated once per process and then cached/pooled,
* This will be replaced by the TOC
{:toc}
-## Built-in File Systems
+## Local File System
-Flink ships with implementations for the following file systems:
+Flink has built-in support for the file system of the local machine, including any NFS or SAN drives mounted into that local file system.
+It can be used by default without additional configuration. Local files are referenced with the *file://* URI scheme.
- - **local**: This file system is used when the scheme is *"file://"*, and it represents the file system of the local machine, including any NFS or SAN drives mounted into that local file system.
+## Pluggable File Systems
- - **S3**: Flink directly provides file systems to talk to Amazon S3 with two alternative implementations, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`. Both implementations are self-contained with no dependency footprint.
-
- - **MapR FS**: The MapR file system *"maprfs://"* is automatically available when the MapR libraries are in the classpath.
-
- - **OpenStack Swift FS**: Flink directly provides a file system to talk to the OpenStack Swift file system, registered under the scheme *"swift://"*.
- The implementation of `flink-swift-fs-hadoop` is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
- To use it when using Flink as a library, add the respective maven dependency (`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`
- When starting a Flink application from the Flink binaries, copy or move the respective jar file from the `opt` folder to the `lib` folder.
+The Apache Flink project supports the following file systems:
+
+ - [**Amazon S3**](./s3.html) object storage is supported by two alternative implementations: `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
+ Both implementations are self-contained with no dependency footprint.
+
+ - **MapR FS** file system adapter is already supported in the main Flink distribution under the *maprfs://* URI scheme.
+ You must provide the MapR libraries in the classpath (for example in `lib` directory).
+
+ - **OpenStack Swift FS** is supported by `flink-swift-fs-hadoop` and registered under the *swift://* URI scheme.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
+ To use it when using Flink as a library, add the respective maven dependency (`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`).
- - **Azure Blob Storage**:
- Flink directly provides a file system to work with Azure Blob Storage.
- This filesystem is registered under the scheme *"wasb(s)://"*.
- The implementation is self-contained with no dependency footprint.
+ - **[Aliyun Object Storage Service](./oss.html)** is supported by `flink-oss-fs-hadoop` and registered under the *oss://* URI scheme.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
-## HDFS and Hadoop File System support
+ - **[Azure Blob Storage](./azure.html)** is supported by `flink-azure-fs-hadoop` and registered under the *wasb(s)://* URI schemes.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
+
+Except **MapR FS**, you can use any of them as plugins.
+
+To use a pluggable file systems, copy the corresponding JAR file from the `opt` directory to a directory under `plugins` directory
+of your Flink distribution before starting Flink, e.g.
+
+{% highlight bash %}
+mkdir ./plugins/s3-fs-hadoop
+cp ./opt/flink-s3-fs-hadoop-{{ site.version }}.jar ./plugins/s3-fs-hadoop/
+{% endhighlight %}
+
+<span class="label label-danger">Attention</span> The plugin mechanism for file systems was introduced in Flink version `1.9` to
+support dedicated Java class loaders per plugin and to move away from the class shading mechanism.
+You can still use the provided file systems (or your own implementations) via the old mechanism by copying the corresponding
+JAR file into `lib` directory.
+
+It's encouraged to use the plugin-based loading mechanism for file systems that support it. Loading file systems components from the `lib`
+directory may be not supported in future Flink versions.
+
+## HDFS and Hadoop File System support
For all schemes where Flink cannot find a directly supported file system, it falls back to Hadoop.
All Hadoop file systems are automatically available when `flink-runtime` and the Hadoop libraries are on the classpath.
-
This way, Flink seamlessly supports all of Hadoop file systems, and all Hadoop-compatible file systems (HCFS).
- **hdfs**
@@ -83,17 +105,21 @@ fs.hdfs.hadoopconf: /path/to/etc/hadoop
This registers `/path/to/etc/hadoop` as Hadoop's configuration directory and is where Flink will look for the `core-site.xml` and `hdfs-site.xml` files.
-## Adding new File System Implementations
+## Adding a new pluggable File System implementation
-File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system.
-Implementations are discovered by Flink through Java's service abstraction, making it easy to add new file system implementations.
+File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system.
To add a new file system:
- Add the File System implementation, which is a subclass of `org.apache.flink.core.fs.FileSystem`.
- Add a factory that instantiates that file system and declares the scheme under which the FileSystem is registered. This must be a subclass of `org.apache.flink.core.fs.FileSystemFactory`.
- - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class.
+ - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class
+ (see the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details).
+
+During plugins discovery, the file system factory class will be loaded by a dedicated Java class loader to avoid class conflicts with other plugins and Flink components.
+The same class loader should be used during file system instantiation and the file system operation calls.
-See the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details on how service loaders work.
+<span class="label label-warning">Warning</span> In practice, it means you should avoid using `Thread.currentThread().getContextClassLoader()` class loader
+in your implementation.
{% top %}
diff --git a/docs/ops/filesystems/index.zh.md b/docs/ops/filesystems/index.zh.md
index 414c82f..88f5133 100644
--- a/docs/ops/filesystems/index.zh.md
+++ b/docs/ops/filesystems/index.zh.md
@@ -25,7 +25,7 @@ under the License.
-->
Apache Flink uses file systems to consume and persistently store data, both for the results of applications and for fault tolerance and recovery.
-These are some of most of the popular file systems, including *local*, *hadoop-compatible*, *S3*, *MapR FS*, *OpenStack Swift FS*, *Aliyun OSS* and *Azure Blob Storage*.
+These are some of most of the popular file systems, including *local*, *hadoop-compatible*, *Amazon S3*, *MapR FS*, *OpenStack Swift FS*, *Aliyun OSS* and *Azure Blob Storage*.
The file system used for a particular file is determined by its URI scheme.
For example, `file:///home/user/text.txt` refers to a file in the local file system, while `hdfs://namenode:50010/data/user/text.txt` is a file in a specific HDFS cluster.
@@ -35,32 +35,54 @@ File system instances are instantiated once per process and then cached/pooled,
* This will be replaced by the TOC
{:toc}
-## Built-in File Systems
+## Local File System
-Flink ships with implementations for the following file systems:
+Flink has built-in support for the file system of the local machine, including any NFS or SAN drives mounted into that local file system.
+It can be used by default without additional configuration. Local files are referenced with the *file://* URI scheme.
- - **local**: This file system is used when the scheme is *"file://"*, and it represents the file system of the local machine, including any NFS or SAN drives mounted into that local file system.
+## Pluggable File Systems
- - **S3**: Flink directly provides file systems to talk to Amazon S3 with two alternative implementations, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`. Both implementations are self-contained with no dependency footprint.
-
- - **MapR FS**: The MapR file system *"maprfs://"* is automatically available when the MapR libraries are in the classpath.
-
- - **OpenStack Swift FS**: Flink directly provides a file system to talk to the OpenStack Swift file system, registered under the scheme *"swift://"*.
- The implementation of `flink-swift-fs-hadoop` is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
- To use it when using Flink as a library, add the respective maven dependency (`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`
- When starting a Flink application from the Flink binaries, copy or move the respective jar file from the `opt` folder to the `lib` folder.
+The Apache Flink project supports the following file systems:
+
+ - [**Amazon S3**](./s3.html) object storage is supported by two alternative implementations: `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
+ Both implementations are self-contained with no dependency footprint.
+
+ - **MapR FS** file system adapter is already supported in the main Flink distribution under the *maprfs://* URI scheme.
+ You must provide the MapR libraries in the classpath (for example in `lib` directory).
+
+ - **OpenStack Swift FS** is supported by `flink-swift-fs-hadoop` and registered under the *swift://* URI scheme.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
+ To use it when using Flink as a library, add the respective maven dependency (`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`).
- - **Azure Blob Storage**:
- Flink directly provides a file system to work with Azure Blob Storage.
- This filesystem is registered under the scheme *"wasb(s)://"*.
- The implementation is self-contained with no dependency footprint.
+ - **[Aliyun Object Storage Service](./oss.html)** is supported by `flink-oss-fs-hadoop` and registered under the *oss://* URI scheme.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
-## HDFS and Hadoop File System support
+ - **[Azure Blob Storage](./azure.html)** is supported by `flink-azure-fs-hadoop` and registered under the *wasb(s)://* URI schemes.
+ The implementation is based on the [Hadoop Project](https://hadoop.apache.org/) but is self-contained with no dependency footprint.
+
+Except **MapR FS**, you can use any of them as plugins.
+
+To use a pluggable file systems, copy the corresponding JAR file from the `opt` directory to a directory under `plugins` directory
+of your Flink distribution before starting Flink, e.g.
+
+{% highlight bash %}
+mkdir ./plugins/s3-fs-hadoop
+cp ./opt/flink-s3-fs-hadoop-{{ site.version }}.jar ./plugins/s3-fs-hadoop/
+{% endhighlight %}
+
+<span class="label label-danger">Attention</span> The plugin mechanism for file systems was introduced in Flink version `1.9` to
+support dedicated Java class loaders per plugin and to move away from the class shading mechanism.
+You can still use the provided file systems (or your own implementations) via the old mechanism by copying the corresponding
+JAR file into `lib` directory.
+
+It's encouraged to use the plugin-based loading mechanism for file systems that support it. Loading file systems components from the `lib`
+directory may be not supported in future Flink versions.
+
+## HDFS and Hadoop File System support
For all schemes where Flink cannot find a directly supported file system, it falls back to Hadoop.
All Hadoop file systems are automatically available when `flink-runtime` and the Hadoop libraries are on the classpath.
-
This way, Flink seamlessly supports all of Hadoop file systems, and all Hadoop-compatible file systems (HCFS).
- **hdfs**
@@ -83,17 +105,21 @@ fs.hdfs.hadoopconf: /path/to/etc/hadoop
This registers `/path/to/etc/hadoop` as Hadoop's configuration directory and is where Flink will look for the `core-site.xml` and `hdfs-site.xml` files.
-## Adding new File System Implementations
+## Adding a new pluggable File System implementation
-File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system.
-Implementations are discovered by Flink through Java's service abstraction, making it easy to add new file system implementations.
+File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system.
To add a new file system:
- Add the File System implementation, which is a subclass of `org.apache.flink.core.fs.FileSystem`.
- Add a factory that instantiates that file system and declares the scheme under which the FileSystem is registered. This must be a subclass of `org.apache.flink.core.fs.FileSystemFactory`.
- - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class.
+ - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class
+ (see the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details).
+
+During plugins discovery, the file system factory class will be loaded by a dedicated Java class loader to avoid class conflicts with other plugins and Flink components.
+The same class loader should be used during file system instantiation and the file system operation calls.
-See the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details on how service loaders work.
+<span class="label label-warning">Warning</span> In practice, it means you should avoid using `Thread.currentThread().getContextClassLoader()` class loader
+in your implementation.
{% top %}
diff --git a/docs/ops/filesystems/oss.md b/docs/ops/filesystems/oss.md
index 0c98c43..e2af733 100644
--- a/docs/ops/filesystems/oss.md
+++ b/docs/ops/filesystems/oss.md
@@ -37,7 +37,7 @@ You can use OSS objects like regular files by specifying paths in the following
oss://<your-bucket>/<object-name>
{% endhighlight %}
-Below shows how to use OSS with Flink:
+Below shows how to use OSS in a Flink job:
{% highlight java %}
// Read from OSS bucket
@@ -50,17 +50,19 @@ stream.writeAsText("oss://<your-bucket>/<object-name>")
env.setStateBackend(new FsStateBackend("oss://<your-bucket>/<object-name>"));
{% endhighlight %}
-### Shaded Hadoop OSS file system
+### Shaded Hadoop OSS file system
-To use `flink-oss-fs-hadoop,` copy the respective JAR file from the opt directory to the lib directory of your Flink distribution before starting Flink, e.g.
+To use `flink-oss-fs-hadoop,` copy the respective JAR file from the `opt` directory to a directory in `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+mkdir ./plugins/oss-fs-hadoop
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./plugins/oss-fs-hadoop/
{% endhighlight %}
-`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the oss:// scheme.
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the *oss://* scheme.
#### Configurations setup
+
After setting up the OSS FileSystem wrapper, you need to add some configurations to make sure that Flink is allowed to access your OSS buckets.
To allow for easy adoption, you can use the same configuration keys in `flink-conf.yaml` as in Hadoop's `core-site.xml`
diff --git a/docs/ops/filesystems/oss.zh.md b/docs/ops/filesystems/oss.zh.md
index d6834d1..f310d20 100644
--- a/docs/ops/filesystems/oss.zh.md
+++ b/docs/ops/filesystems/oss.zh.md
@@ -37,7 +37,7 @@ You can use OSS objects like regular files by specifying paths in the following
oss://<your-bucket>/<object-name>
{% endhighlight %}
-Below shows how to use OSS with Flink:
+Below shows how to use OSS in a Flink job:
{% highlight java %}
// Read from OSS bucket
@@ -50,17 +50,19 @@ stream.writeAsText("oss://<your-bucket>/<object-name>")
env.setStateBackend(new FsStateBackend("oss://<your-bucket>/<object-name>"));
{% endhighlight %}
-### Shaded Hadoop OSS file system
+### Shaded Hadoop OSS file system
-To use `flink-oss-fs-hadoop,` copy the respective JAR file from the opt directory to the lib directory of your Flink distribution before starting Flink, e.g.
+To use `flink-oss-fs-hadoop,` copy the respective JAR file from the `opt` directory to a directory in `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+mkdir ./plugins/oss-fs-hadoop
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./plugins/oss-fs-hadoop/
{% endhighlight %}
-`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the oss:// scheme.
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the *oss://* scheme.
#### Configurations setup
+
After setting up the OSS FileSystem wrapper, you need to add some configurations to make sure that Flink is allowed to access your OSS buckets.
To allow for easy adoption, you can use the same configuration keys in `flink-conf.yaml` as in Hadoop's `core-site.xml`
diff --git a/docs/ops/filesystems/s3.md b/docs/ops/filesystems/s3.md
index f25e266..f601b46 100644
--- a/docs/ops/filesystems/s3.md
+++ b/docs/ops/filesystems/s3.md
@@ -59,23 +59,24 @@ For some cases, however, e.g., for using S3 as YARN's resource storage dir, it m
Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
- - `flink-s3-fs-presto`, registered under the scheme *"s3://"* and *"s3p://"*, is based on code from the [Presto project](https://prestodb.io/).
+ - `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
You can configure it the same way you can [configure the Presto file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration) by placing adding the configurations to your `flink-conf.yaml`. Presto is the recommended file system for checkpointing to S3.
-
- - `flink-s3-fs-hadoop`, registered under *"s3://"* and *"s3a://"*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
+
+ - `flink-s3-fs-hadoop`, registered under *s3://* and *s3a://*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
The file system can be [configured exactly like Hadoop's s3a](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by placing adding the configurations to your `flink-conf.yaml`. Shaded Hadoop is the only S3 file system with support for the [StreamingFileSink]({{ site.baseurl}}/dev/connectors/streamfile_sink.html).
-
+
Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem
-wrappers for URIs with the `s3://` scheme, `flink-s3-fs-hadoop` also registers
-for `s3a://` and `flink-s3-fs-presto` also registers for `s3p://`, so you can
+wrappers for URIs with the *s3://* scheme, `flink-s3-fs-hadoop` also registers
+for *s3a://* and `flink-s3-fs-presto` also registers for *s3p://*, so you can
use this to use both at the same time.
For example, the job uses the [StreamingFileSink]({{ site.baseurl}}/dev/connectors/streamfile_sink.html) which only supports Hadoop, but uses Presto for checkpointing.
-In this case, it is advised to use explicitly *"s3a://"* as a scheme for the sink (Hadoop) and *"s3p://"* for checkpointing (Presto).
-
-To use either `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `lib` directory of your Flink distribution before starting Flink, e.g.
+In this case, it is advised to explicitly use *s3a://* as a scheme for the sink (Hadoop) and *s3p://* for checkpointing (Presto).
+
+To use `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./lib/
+mkdir ./plugins/s3-fs-presto
+cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./plugins/s3-fs-presto/
{% endhighlight %}
#### Configure Access Credentials
@@ -102,7 +103,7 @@ s3.secret-key: your-secret-key
## Configure Non-S3 Endpoint
The S3 Filesystems also support using S3 compliant object stores such as [IBM's Cloud Object Storage](https://www.ibm.com/cloud/object-storage) and [Minio](https://min.io/).
-To do so, configure your endpoint in `flink-conf.yaml`.
+To do so, configure your endpoint in `flink-conf.yaml`.
{% highlight yaml %}
s3.endpoint: your-endpoint-hostname
@@ -133,4 +134,4 @@ The `s3.entropy.key` defines the string in paths that is replaced by the random
If a file system operation does not pass the *"inject entropy"* write option, the entropy key substring is simply removed.
The `s3.entropy.length` defines the number of random alphanumeric characters used for entropy.
-{% top %}
\ No newline at end of file
+{% top %}
diff --git a/docs/ops/filesystems/s3.zh.md b/docs/ops/filesystems/s3.zh.md
index f25e266..f601b46 100644
--- a/docs/ops/filesystems/s3.zh.md
+++ b/docs/ops/filesystems/s3.zh.md
@@ -59,23 +59,24 @@ For some cases, however, e.g., for using S3 as YARN's resource storage dir, it m
Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
- - `flink-s3-fs-presto`, registered under the scheme *"s3://"* and *"s3p://"*, is based on code from the [Presto project](https://prestodb.io/).
+ - `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
You can configure it the same way you can [configure the Presto file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration) by placing adding the configurations to your `flink-conf.yaml`. Presto is the recommended file system for checkpointing to S3.
-
- - `flink-s3-fs-hadoop`, registered under *"s3://"* and *"s3a://"*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
+
+ - `flink-s3-fs-hadoop`, registered under *s3://* and *s3a://*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
The file system can be [configured exactly like Hadoop's s3a](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by placing adding the configurations to your `flink-conf.yaml`. Shaded Hadoop is the only S3 file system with support for the [StreamingFileSink]({{ site.baseurl}}/dev/connectors/streamfile_sink.html).
-
+
Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem
-wrappers for URIs with the `s3://` scheme, `flink-s3-fs-hadoop` also registers
-for `s3a://` and `flink-s3-fs-presto` also registers for `s3p://`, so you can
+wrappers for URIs with the *s3://* scheme, `flink-s3-fs-hadoop` also registers
+for *s3a://* and `flink-s3-fs-presto` also registers for *s3p://*, so you can
use this to use both at the same time.
For example, the job uses the [StreamingFileSink]({{ site.baseurl}}/dev/connectors/streamfile_sink.html) which only supports Hadoop, but uses Presto for checkpointing.
-In this case, it is advised to use explicitly *"s3a://"* as a scheme for the sink (Hadoop) and *"s3p://"* for checkpointing (Presto).
-
-To use either `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `lib` directory of your Flink distribution before starting Flink, e.g.
+In this case, it is advised to explicitly use *s3a://* as a scheme for the sink (Hadoop) and *s3p://* for checkpointing (Presto).
+
+To use `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
{% highlight bash %}
-cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./lib/
+mkdir ./plugins/s3-fs-presto
+cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./plugins/s3-fs-presto/
{% endhighlight %}
#### Configure Access Credentials
@@ -102,7 +103,7 @@ s3.secret-key: your-secret-key
## Configure Non-S3 Endpoint
The S3 Filesystems also support using S3 compliant object stores such as [IBM's Cloud Object Storage](https://www.ibm.com/cloud/object-storage) and [Minio](https://min.io/).
-To do so, configure your endpoint in `flink-conf.yaml`.
+To do so, configure your endpoint in `flink-conf.yaml`.
{% highlight yaml %}
s3.endpoint: your-endpoint-hostname
@@ -133,4 +134,4 @@ The `s3.entropy.key` defines the string in paths that is replaced by the random
If a file system operation does not pass the *"inject entropy"* write option, the entropy key substring is simply removed.
The `s3.entropy.length` defines the number of random alphanumeric characters used for entropy.
-{% top %}
\ No newline at end of file
+{% top %}
[flink] 01/03: [hotfix][docs] Reference file systems documentation
from batch/connectors docs
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a170d9e60321c15fe7d885e643cbdaa250bcda4e
Author: Aleksey Pak <al...@ververica.com>
AuthorDate: Thu Jul 18 15:18:01 2019 +0200
[hotfix][docs] Reference file systems documentation from batch/connectors docs
---
docs/dev/batch/connectors.md | 16 ++++------------
docs/dev/batch/connectors.zh.md | 16 ++++------------
2 files changed, 8 insertions(+), 24 deletions(-)
diff --git a/docs/dev/batch/connectors.md b/docs/dev/batch/connectors.md
index 7d13d1e..2d341d7 100644
--- a/docs/dev/batch/connectors.md
+++ b/docs/dev/batch/connectors.md
@@ -25,18 +25,10 @@ under the License.
* TOC
{:toc}
-## Reading from file systems
-
-Flink has built-in support for the following file systems:
-
-| Filesystem | Scheme | Notes |
-| ------------------------------------- |--------------| ------ |
-| Hadoop Distributed File System (HDFS) | `hdfs://` | All HDFS versions are supported |
-| Amazon S3 | `s3://` | Support through Hadoop file system implementation (see below) |
-| MapR file system | `maprfs://` | The user has to manually place the required jar files in the `lib/` dir |
-| Alluxio | `alluxio://` | Support through Hadoop file system implementation (see below) |
-
+## Reading from and writing to file systems
+The Apache Flink project supports multiple [file systems]({{ site.baseurl }}/ops/filesystems/index.html) that can be used as backing stores
+for input and output connectors.
### Using Hadoop file system implementations
@@ -93,7 +85,7 @@ Also, the serialization framework of Flink is able to handle classes generated f
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
- <version>{{site.version }}</version>
+ <version>{{ site.version }}</version>
</dependency>
{% endhighlight %}
diff --git a/docs/dev/batch/connectors.zh.md b/docs/dev/batch/connectors.zh.md
index a647e8e..e577c37 100644
--- a/docs/dev/batch/connectors.zh.md
+++ b/docs/dev/batch/connectors.zh.md
@@ -25,18 +25,10 @@ under the License.
* TOC
{:toc}
-## Reading from file systems
-
-Flink has built-in support for the following file systems:
-
-| Filesystem | Scheme | Notes |
-| ------------------------------------- |--------------| ------ |
-| Hadoop Distributed File System (HDFS) | `hdfs://` | All HDFS versions are supported |
-| Amazon S3 | `s3://` | Support through Hadoop file system implementation (see below) |
-| MapR file system | `maprfs://` | The user has to manually place the required jar files in the `lib/` dir |
-| Alluxio | `alluxio://` | Support through Hadoop file system implementation (see below) |
-
+## Reading from and writing to file systems
+The Apache Flink project supports multiple [file systems]({{ site.baseurl }}/ops/filesystems/index.html) that can be used as backing stores
+for input and output connectors.
### Using Hadoop file system implementations
@@ -93,7 +85,7 @@ Also, the serialization framework of Flink is able to handle classes generated f
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
- <version>{{site.version }}</version>
+ <version>{{ site.version }}</version>
</dependency>
{% endhighlight %}
[flink] 03/03: [FLINK-12998][docs] Extend debugging_classloading
documentation with plugins loading case
Posted by pn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 57f7958685206b063aadbf0c157c1e2f7614c0c1
Author: Aleksey Pak <al...@ververica.com>
AuthorDate: Fri Jul 19 15:22:40 2019 +0200
[FLINK-12998][docs] Extend debugging_classloading documentation with plugins loading case
---
docs/monitoring/debugging_classloading.md | 55 +++++++++++++++-------------
docs/monitoring/debugging_classloading.zh.md | 55 +++++++++++++++-------------
2 files changed, 58 insertions(+), 52 deletions(-)
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
index 15a2519..76a842e 100644
--- a/docs/monitoring/debugging_classloading.md
+++ b/docs/monitoring/debugging_classloading.md
@@ -28,17 +28,20 @@ under the License.
## Overview of Classloading in Flink
When running Flink applications, the JVM will load various classes over time.
-These classes can be divided into two domains:
+These classes can be divided into three groups based on their origin:
- The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
- in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
+ in Flink's `/lib` folder (the classes of Apache Flink and some dependencies).
+
+ - The **Flink Plugin Components**: The plugins code in folders under Flink's `/plugins` folder. Flink's plugin mechanism will dynamically load them once during startup.
- The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
(via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
-What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
-processes first, and submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
-or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
+As a general rule, whenever you start the Flink processes first and submit jobs later, the job's classes are loaded dynamically.
+If the Flink processes are started together with the job/application, or if the application spawns the Flink components (JobManager, TaskManager, etc.), then all job's classes are in the Java classpath.
+
+Code in plugin components is loaded dynamically once by a dedicated class loader per plugin.
In the following are some more details about the different deployment modes:
@@ -59,7 +62,7 @@ created for an job/application and will contain the job/application's jar files.
**Docker / Kubernetes Sessions**
Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
-behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
+behave like standalone sessions: Flink's code is in the Java classpath, plugin components are loaded dynamically at startup and the job's code is loaded dynamically.
**YARN**
@@ -67,8 +70,8 @@ behave like standalone sessions: Flink's code is in the Java classpath, the job'
YARN classloading differs between single job deployments and sessions:
- When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
- That means that there is *no dynamic classloading* involved in that case.
+ JobManagers are started for that job. Those JVMs have user code classes in the Java classpath.
+ That means that there is *no dynamic classloading* involved in that case for the job.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
classpath. The classes from all jobs that are submitted against the session are loaded dynamically.
@@ -82,37 +85,37 @@ classes are loaded dynamically when the jobs are submitted.
## Inverted Class Loading and ClassLoader Resolution Order
-In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
-(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
-for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+In setups where dynamic classloading is involved (plugin components, Flink jobs in session setups), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *plugin/user code classloader*.
+for loading classes from the plugin or the user-code jar(s). The dynamic ClassLoader has the application classloader as its parent.
-By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
-the parent (application classloader) if the class is not part of the dynamically loaded user code.
+By default, Flink inverts classloading order, meaning it looks into the dynamic classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded code.
-The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+The benefit of inverted classloading is that plugins and jobs can use different library versions than Flink's core itself, which is very
useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
-classes (Flink's core or one of its dependencies can use a different copy than the user code).
+classes (Flink's core or one of its dependencies can use a different copy than the user code or plugin code).
In most cases, this works well and no additional configuration from the user is needed.
-However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
-You can revert back to Java's default mode by configuring the ClassLoader resolution order via
-[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+However, there are cases when the inverted classloading causes problems (see below, ["X cannot be cast to X"](#x-cannot-be-cast-to-x-exceptions)).
+For user code classloading, you can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[`classloader.resolve-order`](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
(from Flink's default `child-first`).
Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
-are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
-[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
-[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+are shared between Flink's core and the plugin/user code or the plugin/user-code facing APIs. The packages for these classes are configured via
+[`classloader.parent-first-patterns-default`](../ops/config.html#classloader-parent-first-patterns-default) and
+[`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional).
To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
-## Avoiding Dynamic Classloading
+## Avoiding Dynamic Classloading for User Code
All components (JobManger, TaskManager, Client, ApplicationMaster, ...) log their classpath setting on startup.
They can be found as part of the environment information at the beginning of the log.
-When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put JAR files
+When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put user code JAR files
directly into the `/lib` folder to make sure they are part of the classpath and not loaded dynamically.
It usually works to put the job's JAR file into the `/lib` directory. The JAR will be part of both the classpath
@@ -125,7 +128,7 @@ used by multiple jobs), it may still be possible to put common libraries to the
for those.
-## Manual Classloading in the Job
+## Manual Classloading in User Code
In some cases, a transformation function, source, or sink needs to manually load classes (dynamically via reflection).
To do that, it needs the classloader that has access to the job's classes.
@@ -149,9 +152,9 @@ The solution here is to either have a setup without any dynamic classloading, or
The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
-## Unloading of Dynamically Loaded Classes
+## Unloading of Dynamically Loaded Classes in User Code
-All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
+All scenarios that involve dynamic user code classloading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).
diff --git a/docs/monitoring/debugging_classloading.zh.md b/docs/monitoring/debugging_classloading.zh.md
index a4bea0e..f41c782 100644
--- a/docs/monitoring/debugging_classloading.zh.md
+++ b/docs/monitoring/debugging_classloading.zh.md
@@ -28,17 +28,20 @@ under the License.
## Overview of Classloading in Flink
When running Flink applications, the JVM will load various classes over time.
-These classes can be divided into two domains:
+These classes can be divided into three groups based on their origin:
- The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
- in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
+ in Flink's `/lib` folder (the classes of Apache Flink and some dependencies).
+
+ - The **Flink Plugin Components**: The plugins code in folders under Flink's `/plugins` folder. Flink's plugin mechanism will dynamically load them once during startup.
- The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
(via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
-What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
-processes first, and submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
-or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
+As a general rule, whenever you start the Flink processes first and submit jobs later, the job's classes are loaded dynamically.
+If the Flink processes are started together with the job/application, or if the application spawns the Flink components (JobManager, TaskManager, etc.), then all job's classes are in the Java classpath.
+
+Code in plugin components is loaded dynamically once by a dedicated class loader per plugin.
In the following are some more details about the different deployment modes:
@@ -59,7 +62,7 @@ created for an job/application and will contain the job/application's jar files.
**Docker / Kubernetes Sessions**
Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
-behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
+behave like standalone sessions: Flink's code is in the Java classpath, plugin components are loaded dynamically at startup and the job's code is loaded dynamically.
**YARN**
@@ -67,8 +70,8 @@ behave like standalone sessions: Flink's code is in the Java classpath, the job'
YARN classloading differs between single job deployments and sessions:
- When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
- That means that there is *no dynamic classloading* involved in that case.
+ JobManagers are started for that job. Those JVMs have user code classes in the Java classpath.
+ That means that there is *no dynamic classloading* involved in that case for the job.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
classpath. The classes from all jobs that are submitted against the session are loaded dynamically.
@@ -82,37 +85,37 @@ classes are loaded dynamically when the jobs are submitted.
## Inverted Class Loading and ClassLoader Resolution Order
-In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
-(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
-for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+In setups where dynamic classloading is involved (plugin components, Flink jobs in session setups), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *plugin/user code classloader*.
+for loading classes from the plugin or the user-code jar(s). The dynamic ClassLoader has the application classloader as its parent.
-By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
-the parent (application classloader) if the class is not part of the dynamically loaded user code.
+By default, Flink inverts classloading order, meaning it looks into the dynamic classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded code.
-The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+The benefit of inverted classloading is that plugins and jobs can use different library versions than Flink's core itself, which is very
useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
-classes (Flink's core or one of its dependencies can use a different copy than the user code).
+classes (Flink's core or one of its dependencies can use a different copy than the user code or plugin code).
In most cases, this works well and no additional configuration from the user is needed.
-However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
-You can revert back to Java's default mode by configuring the ClassLoader resolution order via
-[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+However, there are cases when the inverted classloading causes problems (see below, ["X cannot be cast to X"](#x-cannot-be-cast-to-x-exceptions)).
+For user code classloading, you can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[`classloader.resolve-order`](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
(from Flink's default `child-first`).
Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
-are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
-[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
-[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+are shared between Flink's core and the plugin/user code or the plugin/user-code facing APIs. The packages for these classes are configured via
+[`classloader.parent-first-patterns-default`](../ops/config.html#classloader-parent-first-patterns-default) and
+[`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional).
To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
-## Avoiding Dynamic Classloading
+## Avoiding Dynamic Classloading for User Code
All components (JobManger, TaskManager, Client, ApplicationMaster, ...) log their classpath setting on startup.
They can be found as part of the environment information at the beginning of the log.
-When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put JAR files
+When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put user code JAR files
directly into the `/lib` folder to make sure they are part of the classpath and not loaded dynamically.
It usually works to put the job's JAR file into the `/lib` directory. The JAR will be part of both the classpath
@@ -125,7 +128,7 @@ used by multiple jobs), it may still be possible to put common libraries to the
for those.
-## Manual Classloading in the Job
+## Manual Classloading in User Code
In some cases, a transformation function, source, or sink needs to manually load classes (dynamically via reflection).
To do that, it needs the classloader that has access to the job's classes.
@@ -149,9 +152,9 @@ The solution here is to either have a setup without any dynamic classloading, or
The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
-## Unloading of Dynamically Loaded Classes
+## Unloading of Dynamically Loaded Classes in User Code
-All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
+All scenarios that involve dynamic user code classloading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).