You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by pn...@apache.org on 2019/08/01 08:31:15 UTC
[flink] 03/03: [FLINK-12998][docs] Extend debugging_classloading
documentation with plugins loading case
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 57f7958685206b063aadbf0c157c1e2f7614c0c1
Author: Aleksey Pak <al...@ververica.com>
AuthorDate: Fri Jul 19 15:22:40 2019 +0200
[FLINK-12998][docs] Extend debugging_classloading documentation with plugins loading case
---
docs/monitoring/debugging_classloading.md | 55 +++++++++++++++-------------
docs/monitoring/debugging_classloading.zh.md | 55 +++++++++++++++-------------
2 files changed, 58 insertions(+), 52 deletions(-)
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
index 15a2519..76a842e 100644
--- a/docs/monitoring/debugging_classloading.md
+++ b/docs/monitoring/debugging_classloading.md
@@ -28,17 +28,20 @@ under the License.
## Overview of Classloading in Flink
When running Flink applications, the JVM will load various classes over time.
-These classes can be divided into two domains:
+These classes can be divided into three groups based on their origin:
- The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
- in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
+ in Flink's `/lib` folder (the classes of Apache Flink and some dependencies).
+
+ - The **Flink Plugin Components**: The plugins code in folders under Flink's `/plugins` folder. Flink's plugin mechanism will dynamically load them once during startup.
- The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
(via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
-What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
-processes first, and submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
-or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
+As a general rule, whenever you start the Flink processes first and submit jobs later, the job's classes are loaded dynamically.
+If the Flink processes are started together with the job/application, or if the application spawns the Flink components (JobManager, TaskManager, etc.), then all job's classes are in the Java classpath.
+
+Code in plugin components is loaded dynamically once by a dedicated class loader per plugin.
In the following are some more details about the different deployment modes:
@@ -59,7 +62,7 @@ created for an job/application and will contain the job/application's jar files.
**Docker / Kubernetes Sessions**
Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
-behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
+behave like standalone sessions: Flink's code is in the Java classpath, plugin components are loaded dynamically at startup and the job's code is loaded dynamically.
**YARN**
@@ -67,8 +70,8 @@ behave like standalone sessions: Flink's code is in the Java classpath, the job'
YARN classloading differs between single job deployments and sessions:
- When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
- That means that there is *no dynamic classloading* involved in that case.
+ JobManagers are started for that job. Those JVMs have user code classes in the Java classpath.
+ That means that there is *no dynamic classloading* involved in that case for the job.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
classpath. The classes from all jobs that are submitted against the session are loaded dynamically.
@@ -82,37 +85,37 @@ classes are loaded dynamically when the jobs are submitted.
## Inverted Class Loading and ClassLoader Resolution Order
-In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
-(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
-for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+In setups where dynamic classloading is involved (plugin components, Flink jobs in session setups), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *plugin/user code classloader*.
+for loading classes from the plugin or the user-code jar(s). The dynamic ClassLoader has the application classloader as its parent.
-By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
-the parent (application classloader) if the class is not part of the dynamically loaded user code.
+By default, Flink inverts classloading order, meaning it looks into the dynamic classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded code.
-The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+The benefit of inverted classloading is that plugins and jobs can use different library versions than Flink's core itself, which is very
useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
-classes (Flink's core or one of its dependencies can use a different copy than the user code).
+classes (Flink's core or one of its dependencies can use a different copy than the user code or plugin code).
In most cases, this works well and no additional configuration from the user is needed.
-However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
-You can revert back to Java's default mode by configuring the ClassLoader resolution order via
-[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+However, there are cases when the inverted classloading causes problems (see below, ["X cannot be cast to X"](#x-cannot-be-cast-to-x-exceptions)).
+For user code classloading, you can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[`classloader.resolve-order`](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
(from Flink's default `child-first`).
Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
-are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
-[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
-[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+are shared between Flink's core and the plugin/user code or the plugin/user-code facing APIs. The packages for these classes are configured via
+[`classloader.parent-first-patterns-default`](../ops/config.html#classloader-parent-first-patterns-default) and
+[`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional).
To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
-## Avoiding Dynamic Classloading
+## Avoiding Dynamic Classloading for User Code
All components (JobManger, TaskManager, Client, ApplicationMaster, ...) log their classpath setting on startup.
They can be found as part of the environment information at the beginning of the log.
-When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put JAR files
+When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put user code JAR files
directly into the `/lib` folder to make sure they are part of the classpath and not loaded dynamically.
It usually works to put the job's JAR file into the `/lib` directory. The JAR will be part of both the classpath
@@ -125,7 +128,7 @@ used by multiple jobs), it may still be possible to put common libraries to the
for those.
-## Manual Classloading in the Job
+## Manual Classloading in User Code
In some cases, a transformation function, source, or sink needs to manually load classes (dynamically via reflection).
To do that, it needs the classloader that has access to the job's classes.
@@ -149,9 +152,9 @@ The solution here is to either have a setup without any dynamic classloading, or
The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
-## Unloading of Dynamically Loaded Classes
+## Unloading of Dynamically Loaded Classes in User Code
-All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
+All scenarios that involve dynamic user code classloading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).
diff --git a/docs/monitoring/debugging_classloading.zh.md b/docs/monitoring/debugging_classloading.zh.md
index a4bea0e..f41c782 100644
--- a/docs/monitoring/debugging_classloading.zh.md
+++ b/docs/monitoring/debugging_classloading.zh.md
@@ -28,17 +28,20 @@ under the License.
## Overview of Classloading in Flink
When running Flink applications, the JVM will load various classes over time.
-These classes can be divided into two domains:
+These classes can be divided into three groups based on their origin:
- The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
- in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
+ in Flink's `/lib` folder (the classes of Apache Flink and some dependencies).
+
+ - The **Flink Plugin Components**: The plugins code in folders under Flink's `/plugins` folder. Flink's plugin mechanism will dynamically load them once during startup.
- The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
(via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
-What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
-processes first, and submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
-or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
+As a general rule, whenever you start the Flink processes first and submit jobs later, the job's classes are loaded dynamically.
+If the Flink processes are started together with the job/application, or if the application spawns the Flink components (JobManager, TaskManager, etc.), then all job's classes are in the Java classpath.
+
+Code in plugin components is loaded dynamically once by a dedicated class loader per plugin.
In the following are some more details about the different deployment modes:
@@ -59,7 +62,7 @@ created for an job/application and will contain the job/application's jar files.
**Docker / Kubernetes Sessions**
Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
-behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
+behave like standalone sessions: Flink's code is in the Java classpath, plugin components are loaded dynamically at startup and the job's code is loaded dynamically.
**YARN**
@@ -67,8 +70,8 @@ behave like standalone sessions: Flink's code is in the Java classpath, the job'
YARN classloading differs between single job deployments and sessions:
- When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
- That means that there is *no dynamic classloading* involved in that case.
+ JobManagers are started for that job. Those JVMs have user code classes in the Java classpath.
+ That means that there is *no dynamic classloading* involved in that case for the job.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
classpath. The classes from all jobs that are submitted against the session are loaded dynamically.
@@ -82,37 +85,37 @@ classes are loaded dynamically when the jobs are submitted.
## Inverted Class Loading and ClassLoader Resolution Order
-In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
-(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
-for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+In setups where dynamic classloading is involved (plugin components, Flink jobs in session setups), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *plugin/user code classloader*.
+for loading classes from the plugin or the user-code jar(s). The dynamic ClassLoader has the application classloader as its parent.
-By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
-the parent (application classloader) if the class is not part of the dynamically loaded user code.
+By default, Flink inverts classloading order, meaning it looks into the dynamic classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded code.
-The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+The benefit of inverted classloading is that plugins and jobs can use different library versions than Flink's core itself, which is very
useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
-classes (Flink's core or one of its dependencies can use a different copy than the user code).
+classes (Flink's core or one of its dependencies can use a different copy than the user code or plugin code).
In most cases, this works well and no additional configuration from the user is needed.
-However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
-You can revert back to Java's default mode by configuring the ClassLoader resolution order via
-[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+However, there are cases when the inverted classloading causes problems (see below, ["X cannot be cast to X"](#x-cannot-be-cast-to-x-exceptions)).
+For user code classloading, you can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[`classloader.resolve-order`](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
(from Flink's default `child-first`).
Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
-are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
-[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
-[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+are shared between Flink's core and the plugin/user code or the plugin/user-code facing APIs. The packages for these classes are configured via
+[`classloader.parent-first-patterns-default`](../ops/config.html#classloader-parent-first-patterns-default) and
+[`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional).
To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
-## Avoiding Dynamic Classloading
+## Avoiding Dynamic Classloading for User Code
All components (JobManger, TaskManager, Client, ApplicationMaster, ...) log their classpath setting on startup.
They can be found as part of the environment information at the beginning of the log.
-When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put JAR files
+When running a setup where the Flink JobManager and TaskManagers are exclusive to one particular job, one can put user code JAR files
directly into the `/lib` folder to make sure they are part of the classpath and not loaded dynamically.
It usually works to put the job's JAR file into the `/lib` directory. The JAR will be part of both the classpath
@@ -125,7 +128,7 @@ used by multiple jobs), it may still be possible to put common libraries to the
for those.
-## Manual Classloading in the Job
+## Manual Classloading in User Code
In some cases, a transformation function, source, or sink needs to manually load classes (dynamically via reflection).
To do that, it needs the classloader that has access to the job's classes.
@@ -149,9 +152,9 @@ The solution here is to either have a setup without any dynamic classloading, or
The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
-## Unloading of Dynamically Loaded Classes
+## Unloading of Dynamically Loaded Classes in User Code
-All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
+All scenarios that involve dynamic user code classloading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).