You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2018/05/24 18:03:38 UTC
[1/6] flink git commit: [FLINK-9286] [docs] Update classloading docs
Repository: flink
Updated Branches:
refs/heads/master c47cc87e5 -> 33aee8069
[FLINK-9286] [docs] Update classloading docs
This closes #5948
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/428a900c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/428a900c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/428a900c
Branch: refs/heads/master
Commit: 428a900ce518297bb2fbf3421110594bc1e861c9
Parents: a202d74
Author: zentol <ch...@apache.org>
Authored: Wed May 2 12:02:53 2018 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:52 2018 +0200
----------------------------------------------------------------------
docs/monitoring/debugging_classloading.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/428a900c/docs/monitoring/debugging_classloading.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
index 187ffaf..152494a 100644
--- a/docs/monitoring/debugging_classloading.md
+++ b/docs/monitoring/debugging_classloading.md
@@ -69,10 +69,10 @@ cases. By default, Java ClassLoaders will first look for classes in the parent C
the child ClassLoader for cases where we have a hierarchy of ClassLoaders. This is problematic if you
have in your user jar a version of a library that conflicts with a version that comes with Flink. You can
change this behaviour by configuring the ClassLoader resolution order via
-`classloader.resolve-order: child-first` in the Flink config. However, Flink classes will still
-be resolved through the parent ClassLoader first, although you can also configure this via
-`classloader.parent-first-patterns` (see [config](../ops/config.html))
-
+[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config.
+Note that certain classes are always resolved through the parent ClassLoader first, which can be configured via
+[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
+[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
## Avoiding Dynamic Classloading
[6/6] flink git commit: [FLINK-9286] [docs] (addendum) Update
classloading docs for inverted classloading
Posted by se...@apache.org.
[FLINK-9286] [docs] (addendum) Update classloading docs for inverted classloading
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/c07b693f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/c07b693f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/c07b693f
Branch: refs/heads/master
Commit: c07b693fe3e39ed11deef89151aff83729a74268
Parents: 428a900
Author: Stephan Ewen <se...@apache.org>
Authored: Thu May 24 17:05:24 2018 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:53 2018 +0200
----------------------------------------------------------------------
docs/monitoring/debugging_classloading.md | 109 ++++++++++++++++---------
1 file changed, 69 insertions(+), 40 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/c07b693f/docs/monitoring/debugging_classloading.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
index 152494a..fe2c36d 100644
--- a/docs/monitoring/debugging_classloading.md
+++ b/docs/monitoring/debugging_classloading.md
@@ -30,26 +30,44 @@ under the License.
When running Flink applications, the JVM will load various classes over time.
These classes can be divided into two domains:
- - The **Flink Framework** domain: This includes all code in the `/lib` directory in the Flink directory.
- By default these are the classes of Apache Flink and its core dependencies.
+ - The **Java Classpath**: This is Java's common classpath, and it includes the JDK libraries, and all code
+ in Flink's `/lib` folder (the classes of Apache Flink and its core dependencies).
- - The **User Code** domain: These are all classes that are included in the JAR file submitted via the CLI or web interface.
- That includes the job's classes, and all libraries and connectors that are put into the uber JAR.
+ - The **Dynamic User Code**: These are all classes that are included in the JAR files of dynamically submitted jobs,
+ (via REST, CLI, web UI). They are loaded (and unloaded) dynamically per job.
+What classes are part of which domain depends on the particular setup in which you run Apache Flink. As a general rule, whenever you start the Flink
+processes first, and the submit jobs, the job's classes are loaded dynamically. If the Flink processes are started together with the job/application,
+or the application spawns the Flink components (JobManager, TaskManager, etc.) then all classes are in the Java classpath.
-The class loading behaves slightly different for various Flink setups:
+In the following are some more details about the different deployment modes:
-**Standalone**
+**Standalone Session**
+
+When starting a Flink cluster as a standalone session, the JobManagers and TaskManagers are started with the Flink framework classes in the
+Java classpath. The classes from all jobs/applications that are submitted against the session (via REST / CLI) are loaded *dynamically*.
+
+<!--
+**Docker Containers with Flink-as-a-Library**
+
+If you package a Flink job/application such that your application treats Flink like a library (Flink JobManager/TaskManager daemons as spawned as needed),
+then typically all classes are in the *application classpath*. This is the recommended way for container-based setups where the container is specifically
+created for an job/application and will contain the job/application's jar files.
+
+-->
+
+**Docker / Kubernetes Sessions**
+
+Docker / Kubernetes setups that start first a set of JobManagers / TaskManagers and then submit jobs/applications via REST or the CLI
+behave like standalone sessions: Flink's code is in the Java classpath, the job's code is loaded dynamically.
-When starting a Flink cluster, the JobManagers and TaskManagers are started with the Flink framework classes in the
-classpath. The classes from all jobs that are submitted against the cluster are loaded *dynamically*.
**YARN**
YARN classloading differs between single job deployments and sessions:
- - When submitting a Flink job directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
- JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in their classpath.
+ - When submitting a Flink job/application directly to YARN (via `bin/flink run -m yarn-cluster ...`), dedicated TaskManagers and
+ JobManagers are started for that job. Those JVMs have both Flink framework classes and user code classes in the Java classpath.
That means that there is *no dynamic classloading* involved in that case.
- When starting a YARN session, the JobManagers and TaskManagers are started with the Flink framework classes in the
@@ -58,21 +76,37 @@ YARN classloading differs between single job deployments and sessions:
**Mesos**
Mesos setups following [this documentation](../ops/deployment/mesos.html) currently behave very much like the a
-YARN session: The TaskManager and JobManager processes are started with the Flink framework classes in classpath, job
+YARN session: The TaskManager and JobManager processes are started with the Flink framework classes in the Java classpath, job
classes are loaded dynamically when the jobs are submitted.
-## Configuring ClassLoader Resolution Order
-Flink uses a hierarchy of ClassLoaders for loading classes from the user-code jar(s). The user-code
-ClassLoader has a reference to the parent ClassLoader, which is the default Java ClassLoader in most
-cases. By default, Java ClassLoaders will first look for classes in the parent ClassLoader and then in
-the child ClassLoader for cases where we have a hierarchy of ClassLoaders. This is problematic if you
-have in your user jar a version of a library that conflicts with a version that comes with Flink. You can
-change this behaviour by configuring the ClassLoader resolution order via
-[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config.
-Note that certain classes are always resolved through the parent ClassLoader first, which can be configured via
+## Inverted Class Loading and ClassLoader Resolution Order
+
+In setups where dynamic classloading is involved (sessions), there is a hierarchy of typically two ClassLoaders:
+(1) Java's *application classloader*, which has all classes in the classpath, and (2) the dynamic *user code classloader*.
+for loading classes from the user-code jar(s). The user-code ClassLoader has the application classloader as its parent.
+cases.
+
+By default, Flink inverts classloading order, meaning it looks into the user code classloader first, and only looks into
+the parent (application classloader) if the class is not part of the dynamically loaded user code.
+
+The benefit of inverted classloading is that jobs can use different library versions than Flink's core itself, which is very
+useful when the different versions of the libraries are not compatible. The mechanism helps to avoid the common dependency conflict
+errors like `IllegalAccessError` or `NoSuchMethodError`. Different parts of the code simply have separate copies of the
+classes (Flink's core or one of its dependencies can use a different copy than the user code).
+In most cases, this work well and no additional configuration from the user is needed.
+
+However, there are cases when the inverted classloading causes problems (see below, "X cannot be cast to X").
+You can revert back to Java's default mode by configuring the ClassLoader resolution order via
+[classloader.resolve-order](../ops/config.html#classloader-resolve-order) in the Flink config to `parent-first`
+(from Flink's default `child-first`).
+
+Please note that certain classes are always resolved in a *parent-first* way (through the parent ClassLoader first), because they
+are shared between Flink's core and the user code or the user-code facing APIs. The packages for these classes are configured via
[classloader.parent-first-patterns-default](../ops/config.html#classloader-parent-first-patterns-default) and
[classloader.parent-first-patterns-additional](../ops/config.html#classloader-parent-first-patterns-additional).
+To add new packages to be *parent-first* loaded, please set the `classloader.parent-first-patterns-additional` config option.
+
## Avoiding Dynamic Classloading
@@ -103,26 +137,22 @@ and access the user code class loader via `getRuntimeContext().getUserCodeClassL
## X cannot be cast to X exceptions
-When you see an exception in the style `com.foo.X cannot be cast to com.foo.X`, it means that multiple versions of the class
-`com.foo.X` have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.
-
-The reason is in most cases that an object of the `com.foo.X` class loaded from a previous execution attempt is still cached somewhere,
-and picked up by a restarted task/operator that reloaded the code. Note that this is again only possible in deployments that use
-dynamic class loading.
+In setups with dynamic classloading, you may see an exception in the style `com.foo.X cannot be cast to com.foo.X`.
+This means that multiple versions of the class `com.foo.X` have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.
-Common causes of cached object instances:
+One common reason is that a library is not compatible with Flink's *inverted classloading* approach. You can turn off inverted classloading
+to verify this (set [`classloader.resolve-order: parent-first`](../ops/config.html#classloader-resolve-order) in the Flink config) or exclude
+the library from inverted classloading (set [`classloader.parent-first-patterns-additional`](../ops/config.html#classloader-parent-first-patterns-additional)
+in the Flink config).
- - When using *Apache Avro*: The *SpecificDatumReader* caches instances of records. Avoid using `SpecificData.INSTANCE`. See also
- [this discussion](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-get-help-on-ClassCastException-when-re-submitting-a-job-tp10972p11133.html)
-
- - Using certain serialization frameworks for cloning objects (such as *Apache Avro*)
-
- - Interning objects (for example via Guava's Interners)
+Another cause can be cached object instances, as produced by some libraries like *Apache Avro*, or by interning objects (for example via Guava's Interners).
+The solution here is to either have a setup without any dynamic classloading, or to make sure that the respective library is fully part of the dynamically loaded code.
+The latter means that the library must not be added to Flink's `/lib` folder, but must be part of the application's fat-jar/uber-jar
## Unloading of Dynamically Loaded Classes
-All scenarios that involve dynamic class loading (i.e., standalone, sessions, mesos, ...) rely on classes being *unloaded* again.
+All scenarios that involve dynamic class loading (sessions) rely on classes being *unloaded* again.
Class unloading means that the Garbage Collector finds that no objects from a class exist and more, and thus removes the class
(the code, static variable, metadata, etc).
@@ -141,15 +171,14 @@ Common causes for class leaks and suggested fixes:
## Resolving Dependency Conflicts with Flink using the maven-shade-plugin.
-Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often `IllegalAccessExceptions` or `NoSuchMethodError` are the result.
+A way to address dependency conflicts from the application developer's side is to avoid exposing dependencies by *shading them away*.
-Through Hadoop, Flink for example depends on the `aws-sdk` library or on `protobuf-java`. If your user code is using these libraries and you run into issues we recommend relocating the dependency in your user code jar.
-
-Apache Maven offers the [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-plugin/), which allows one to change the package of a class *after* compiling it (so the code you are writing is not affected by the shading). For example if you have the `com.amazonaws` packages from the aws sdk in your user code jar, the shade plugin would relocate them into the `org.myorg.shaded.com.amazonaws` package, so that your code is calling your aws sdk version.
+Apache Maven offers the [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-plugin/), which allows one to change the package of a
+class *after* compiling it (so the code you are writing is not affected by the shading). For example if you have the `com.amazonaws` packages from
+the aws sdk in your user code jar, the shade plugin would relocate them into the `org.myorg.shaded.com.amazonaws` package, so that your code is calling your aws sdk version.
This documentation page explains [relocating classes using the shade plugin](https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html).
-
-Note that some of Flink's dependencies, such as `guava` are shaded away by the maintainers of Flink, so users usually don't have to worry about it.
+Note that most of Flink's dependencies, such as `guava`, `netty`, `jackson`, etc. are shaded away by the maintainers of Flink, so users usually don't have to worry about it.
{% top %}
[5/6] flink git commit: [FLINK-9305] [s3] Also register
flink-s3-fs-hadoop's factory for the s3a:// scheme
Posted by se...@apache.org.
[FLINK-9305] [s3] Also register flink-s3-fs-hadoop's factory for the s3a:// scheme
This closes #5963
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/33aee806
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/33aee806
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/33aee806
Branch: refs/heads/master
Commit: 33aee80694aeeb258cc47aa112505e9ceb993bc2
Parents: 92b0756
Author: Nico Kruber <ni...@data-artisans.com>
Authored: Mon May 7 12:03:15 2018 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:53 2018 +0200
----------------------------------------------------------------------
docs/ops/deployment/aws.md | 2 ++
flink-filesystems/flink-s3-fs-hadoop/README.md | 2 +-
.../flink/fs/s3hadoop/S3AFileSystemFactory.java | 29 ++++++++++++++++++++
.../org.apache.flink.core.fs.FileSystemFactory | 3 +-
.../fs/s3hadoop/HadoopS3FileSystemITCase.java | 23 +++++++++++++---
tools/travis_mvn_watchdog.sh | 23 +++++++++++-----
6 files changed, 69 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/docs/ops/deployment/aws.md
----------------------------------------------------------------------
diff --git a/docs/ops/deployment/aws.md b/docs/ops/deployment/aws.md
index db69b2e..e62e6a3 100644
--- a/docs/ops/deployment/aws.md
+++ b/docs/ops/deployment/aws.md
@@ -105,6 +105,8 @@ To use either `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective
cp ./opt/flink-s3-fs-presto-{{ site.version }}.jar ./lib/
{% endhighlight %}
+Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem wrappers for URIs with the `s3://` scheme, `flink-s3-fs-hadoop` also registers for `s3a://`.
+
#### Configure Access Credentials
After setting up the S3 FileSystem wrapper, you need to make sure that Flink is allowed to access your S3 buckets.
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/flink-filesystems/flink-s3-fs-hadoop/README.md
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-hadoop/README.md b/flink-filesystems/flink-s3-fs-hadoop/README.md
index 11980e0..f65ee61 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/README.md
+++ b/flink-filesystems/flink-s3-fs-hadoop/README.md
@@ -30,7 +30,7 @@ steps are required to keep the shading correct:
2. verify the shaded jar:
- does not contain any unshaded classes except for `org.apache.flink.fs.s3hadoop.S3FileSystemFactory`
- all other classes should be under `org.apache.flink.fs.s3hadoop.shaded`
- - there should be a `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` file pointing to the `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` class
+ - there should be a `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` file pointing to two classes: `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` and `org.apache.flink.fs.s3hadoop.S3AFileSystemFactory`
- other service files under `META-INF/services` should have their names and contents in the relocated `org.apache.flink.fs.s3hadoop.shaded` package
- contains a `core-default-shaded.xml` file
- does not contain a `core-default.xml` or `core-site.xml` file
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3AFileSystemFactory.java
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3AFileSystemFactory.java b/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3AFileSystemFactory.java
new file mode 100644
index 0000000..7f3a068
--- /dev/null
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3AFileSystemFactory.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.s3hadoop;
+
+/**
+ * Simple factory for the S3 file system, registered for the <tt>s3a://</tt> scheme.
+ */
+public class S3AFileSystemFactory extends S3FileSystemFactory {
+ @Override
+ public String getScheme() {
+ return "s3a";
+ }
+}
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory b/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory
index f16986c..b0f723d 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory
@@ -13,4 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.flink.fs.s3hadoop.S3FileSystemFactory
\ No newline at end of file
+org.apache.flink.fs.s3hadoop.S3FileSystemFactory
+org.apache.flink.fs.s3hadoop.S3AFileSystemFactory
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java
index e640603..8672ca4 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemITCase.java
@@ -31,12 +31,16 @@ import org.junit.AfterClass;
import org.junit.Assume;
import org.junit.BeforeClass;
import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
import java.util.UUID;
import static org.apache.flink.core.fs.FileSystemTestUtils.checkPathEventualExistence;
@@ -53,12 +57,19 @@ import static org.junit.Assert.fail;
* <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel">consistency guarantees</a>
* and what the {@link org.apache.hadoop.fs.s3a.S3AFileSystem} offers.
*/
+@RunWith(Parameterized.class)
public class HadoopS3FileSystemITCase extends TestLogger {
+ @Parameterized.Parameter
+ public String scheme;
- private static final String BUCKET = System.getenv("ARTIFACTS_AWS_BUCKET");
+ @Parameterized.Parameters(name = "Scheme = {0}")
+ public static List<String> parameters() {
+ return Arrays.asList("s3", "s3a");
+ }
private static final String TEST_DATA_DIR = "tests-" + UUID.randomUUID();
+ private static final String BUCKET = System.getenv("ARTIFACTS_AWS_BUCKET");
private static final String ACCESS_KEY = System.getenv("ARTIFACTS_AWS_ACCESS_KEY");
private static final String SECRET_KEY = System.getenv("ARTIFACTS_AWS_SECRET_KEY");
@@ -117,9 +128,13 @@ public class HadoopS3FileSystemITCase extends TestLogger {
}
}
+ private String getBasePath() {
+ return scheme + "://" + BUCKET + '/' + TEST_DATA_DIR + "/" + scheme;
+ }
+
@Test
public void testConfigKeysForwarding() throws Exception {
- final Path path = new Path("s3://" + BUCKET + '/');
+ final Path path = new Path(getBasePath());
// access without credentials should fail
{
@@ -181,7 +196,7 @@ public class HadoopS3FileSystemITCase extends TestLogger {
FileSystem.initialize(conf);
- final Path path = new Path("s3://" + BUCKET + '/' + TEST_DATA_DIR + "/test.txt");
+ final Path path = new Path(getBasePath() + "/test.txt");
final FileSystem fs = path.getFileSystem();
try {
@@ -217,7 +232,7 @@ public class HadoopS3FileSystemITCase extends TestLogger {
FileSystem.initialize(conf);
- final Path directory = new Path("s3://" + BUCKET + '/' + TEST_DATA_DIR + "/testdir/");
+ final Path directory = new Path(getBasePath() + "/testdir/");
final FileSystem fs = directory.getFileSystem();
// directory must not yet exist
http://git-wip-us.apache.org/repos/asf/flink/blob/33aee806/tools/travis_mvn_watchdog.sh
----------------------------------------------------------------------
diff --git a/tools/travis_mvn_watchdog.sh b/tools/travis_mvn_watchdog.sh
index e08f0cc..23697d6 100755
--- a/tools/travis_mvn_watchdog.sh
+++ b/tools/travis_mvn_watchdog.sh
@@ -412,7 +412,7 @@ check_shaded_artifacts_s3_fs() {
UNSHADED_CLASSES=`cat allClasses | grep -v -e '^META-INF' -e '^assets' -e "^org/apache/flink/fs/s3${VARIANT}/" | grep '\.class$'`
if [ "$?" == "0" ]; then
echo "=============================================================================="
- echo "Detected unshaded dependencies in fat jar:"
+ echo "${VARIANT}: Detected unshaded dependencies in fat jar:"
echo "${UNSHADED_CLASSES}"
echo "=============================================================================="
return 1
@@ -420,24 +420,33 @@ check_shaded_artifacts_s3_fs() {
if [ ! `cat allClasses | grep '^META-INF/services/org\.apache\.flink\.core\.fs\.FileSystemFactory$'` ]; then
echo "=============================================================================="
- echo "File does not exist: services/org.apache.flink.core.fs.FileSystemFactory"
+ echo "${VARIANT}: File does not exist: services/org.apache.flink.core.fs.FileSystemFactory"
echo "=============================================================================="
+ return 1
fi
UNSHADED_SERVICES=`cat allClasses | grep '^META-INF/services/' | grep -v -e '^META-INF/services/org\.apache\.flink\.core\.fs\.FileSystemFactory$' -e "^META-INF/services/org\.apache\.flink\.fs\.s3${VARIANT}\.shaded" -e '^META-INF/services/'`
if [ "$?" == "0" ]; then
echo "=============================================================================="
- echo "Detected unshaded service files in fat jar:"
+ echo "${VARIANT}: Detected unshaded service files in fat jar:"
echo "${UNSHADED_SERVICES}"
echo "=============================================================================="
return 1
fi
- FS_SERVICE_FILE_CLASS=`unzip -q -c flink-filesystems/flink-s3-fs-${VARIANT}/target/flink-s3-fs-${VARIANT}*.jar META-INF/services/org.apache.flink.core.fs.FileSystemFactory | grep -v -e '^#' -e '^$'`
- if [ "${FS_SERVICE_FILE_CLASS}" != "org.apache.flink.fs.s3${VARIANT}.S3FileSystemFactory" ]; then
+ FS_SERVICE_FILE_CLASSES=`unzip -q -c flink-filesystems/flink-s3-fs-${VARIANT}/target/flink-s3-fs-${VARIANT}*.jar META-INF/services/org.apache.flink.core.fs.FileSystemFactory | grep -v -e '^#' -e '^$'`
+ EXPECTED_FS_SERVICE_FILE_CLASSES="org.apache.flink.fs.s3${VARIANT}.S3FileSystemFactory"
+ if [ "${VARIANT}" == "hadoop" ]; then
+ read -r -d '' EXPECTED_FS_SERVICE_FILE_CLASSES <<EOF
+org.apache.flink.fs.s3${VARIANT}.S3FileSystemFactory
+org.apache.flink.fs.s3${VARIANT}.S3AFileSystemFactory
+EOF
+ fi
+
+ if [ "${FS_SERVICE_FILE_CLASSES}" != "${EXPECTED_FS_SERVICE_FILE_CLASSES}" ]; then
echo "=============================================================================="
- echo "Detected wrong content in services/org.apache.flink.core.fs.FileSystemFactory:"
- echo "${FS_SERVICE_FILE_CLASS}"
+ echo "${VARIANT}: Detected wrong content in services/org.apache.flink.core.fs.FileSystemFactory:"
+ echo "${FS_SERVICE_FILE_CLASSES}"
echo "=============================================================================="
return 1
fi
[4/6] flink git commit: [hotfix] [s3] Port config key forwarding test
from hadoop S3 to presto
Posted by se...@apache.org.
[hotfix] [s3] Port config key forwarding test from hadoop S3 to presto
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/92b0756b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/92b0756b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/92b0756b
Branch: refs/heads/master
Commit: 92b0756bc741f8e6d757cd23551573cd750b9bfc
Parents: 28972d5
Author: Nico Kruber <ni...@data-artisans.com>
Authored: Mon May 7 14:31:47 2018 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:53 2018 +0200
----------------------------------------------------------------------
.../fs/s3presto/PrestoS3FileSystemITCase.java | 68 ++++++++++++++++++++
1 file changed, 68 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/92b0756b/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/PrestoS3FileSystemITCase.java
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/PrestoS3FileSystemITCase.java b/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/PrestoS3FileSystemITCase.java
index caf1636..0783574 100644
--- a/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/PrestoS3FileSystemITCase.java
+++ b/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/PrestoS3FileSystemITCase.java
@@ -32,16 +32,19 @@ import org.junit.BeforeClass;
import org.junit.Test;
import java.io.BufferedReader;
+import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.nio.charset.StandardCharsets;
import java.util.UUID;
+import static com.facebook.presto.hive.PrestoS3FileSystem.S3_USE_INSTANCE_CREDENTIALS;
import static org.apache.flink.core.fs.FileSystemTestUtils.checkPathEventualExistence;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
/**
* Unit tests for the S3 file system support via Presto's {@link com.facebook.presto.hive.PrestoS3FileSystem}.
@@ -67,6 +70,71 @@ public class PrestoS3FileSystemITCase extends TestLogger {
}
@Test
+ public void testConfigKeysForwarding() throws Exception {
+ final Path path = new Path("s3://" + BUCKET + '/' + TEST_DATA_DIR);
+
+ // access without credentials should fail
+ {
+ Configuration conf = new Configuration();
+ // fail fast and do not fall back to trying EC2 credentials
+ conf.setString(S3_USE_INSTANCE_CREDENTIALS, "false");
+ FileSystem.initialize(conf);
+
+ try {
+ path.getFileSystem().exists(path);
+ fail("should fail with an exception");
+ } catch (IOException ignored) {}
+ }
+
+ // standard Presto-style credential keys
+ {
+ Configuration conf = new Configuration();
+ conf.setString(S3_USE_INSTANCE_CREDENTIALS, "false");
+ conf.setString("presto.s3.access-key", ACCESS_KEY);
+ conf.setString("presto.s3.secret-key", SECRET_KEY);
+
+ FileSystem.initialize(conf);
+ path.getFileSystem().exists(path);
+ }
+
+ // shortened Presto-style credential keys
+ {
+ Configuration conf = new Configuration();
+ conf.setString(S3_USE_INSTANCE_CREDENTIALS, "false");
+ conf.setString("s3.access-key", ACCESS_KEY);
+ conf.setString("s3.secret-key", SECRET_KEY);
+
+ FileSystem.initialize(conf);
+ path.getFileSystem().exists(path);
+ }
+
+ // shortened Hadoop-style credential keys
+ {
+ Configuration conf = new Configuration();
+ conf.setString(S3_USE_INSTANCE_CREDENTIALS, "false");
+ conf.setString("s3.access.key", ACCESS_KEY);
+ conf.setString("s3.secret.key", SECRET_KEY);
+
+ FileSystem.initialize(conf);
+ path.getFileSystem().exists(path);
+ }
+
+ // shortened Hadoop-style credential keys with presto prefix
+ {
+ Configuration conf = new Configuration();
+ conf.setString(S3_USE_INSTANCE_CREDENTIALS, "false");
+ conf.setString("presto.s3.access.key", ACCESS_KEY);
+ conf.setString("presto.s3.secret.key", SECRET_KEY);
+
+ FileSystem.initialize(conf);
+ path.getFileSystem().exists(path);
+ }
+
+ // re-set configuration
+ FileSystem.initialize(new Configuration());
+ }
+
+ @Test
public void testSimpleFileWriteAndRead() throws Exception {
final long deadline = System.nanoTime() + 30_000_000_000L; // 30 secs
final Configuration conf = new Configuration();
[3/6] flink git commit: [hotfix] [docs] Fix flink-s3-fs-* README
entry for the service file
Posted by se...@apache.org.
[hotfix] [docs] Fix flink-s3-fs-* README entry for the service file
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/28972d53
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/28972d53
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/28972d53
Branch: refs/heads/master
Commit: 28972d53bb5fbed888495d110c12182bde4a0113
Parents: c07b693
Author: Nico Kruber <ni...@data-artisans.com>
Authored: Mon May 7 12:02:38 2018 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:53 2018 +0200
----------------------------------------------------------------------
flink-filesystems/flink-s3-fs-hadoop/README.md | 2 +-
flink-filesystems/flink-s3-fs-presto/README.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/28972d53/flink-filesystems/flink-s3-fs-hadoop/README.md
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-hadoop/README.md b/flink-filesystems/flink-s3-fs-hadoop/README.md
index 0d61ef3..11980e0 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/README.md
+++ b/flink-filesystems/flink-s3-fs-hadoop/README.md
@@ -30,7 +30,7 @@ steps are required to keep the shading correct:
2. verify the shaded jar:
- does not contain any unshaded classes except for `org.apache.flink.fs.s3hadoop.S3FileSystemFactory`
- all other classes should be under `org.apache.flink.fs.s3hadoop.shaded`
- - there should be a `META-INF/services/org.apache.flink.fs.s3hadoop.S3FileSystemFactory` file pointing to the `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` class
+ - there should be a `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` file pointing to the `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` class
- other service files under `META-INF/services` should have their names and contents in the relocated `org.apache.flink.fs.s3hadoop.shaded` package
- contains a `core-default-shaded.xml` file
- does not contain a `core-default.xml` or `core-site.xml` file
http://git-wip-us.apache.org/repos/asf/flink/blob/28972d53/flink-filesystems/flink-s3-fs-presto/README.md
----------------------------------------------------------------------
diff --git a/flink-filesystems/flink-s3-fs-presto/README.md b/flink-filesystems/flink-s3-fs-presto/README.md
index 9a71abf..7244f5b 100644
--- a/flink-filesystems/flink-s3-fs-presto/README.md
+++ b/flink-filesystems/flink-s3-fs-presto/README.md
@@ -31,7 +31,7 @@ steps are required to keep the shading correct:
2. verify the shaded jar:
- does not contain any unshaded classes except for `org.apache.flink.fs.s3presto.S3FileSystemFactory`
- all other classes should be under `org.apache.flink.fs.s3presto.shaded`
- - there should be a `META-INF/services/org.apache.flink.fs.s3presto.S3FileSystemFactory` file pointing to the `org.apache.flink.fs.s3presto.S3FileSystemFactory` class
+ - there should be a `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` file pointing to the `org.apache.flink.fs.s3presto.S3FileSystemFactory` class
- other service files under `META-INF/services` should have their names and contents in the relocated `org.apache.flink.fs.s3presto.shaded` package
- contains a `core-default-shaded.xml` file
- does not contain a `core-default.xml` or `core-site.xml` file
[2/6] flink git commit: [FLINK-9088] [nifi-connector] Bump
nifi-site-to-site-client to 1.6.0
Posted by se...@apache.org.
[FLINK-9088] [nifi-connector] Bump nifi-site-to-site-client to 1.6.0
This closes #5891
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a202d747
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a202d747
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a202d747
Branch: refs/heads/master
Commit: a202d747cbd7c49ecd4f8f01b4007350c4327b94
Parents: c47cc87
Author: zhouhai02 <zh...@meituan.com>
Authored: Sun Apr 22 15:55:51 2018 +0800
Committer: Stephan Ewen <se...@apache.org>
Committed: Thu May 24 20:02:52 2018 +0200
----------------------------------------------------------------------
flink-connectors/flink-connector-nifi/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/a202d747/flink-connectors/flink-connector-nifi/pom.xml
----------------------------------------------------------------------
diff --git a/flink-connectors/flink-connector-nifi/pom.xml b/flink-connectors/flink-connector-nifi/pom.xml
index dfc73f9..a237673 100644
--- a/flink-connectors/flink-connector-nifi/pom.xml
+++ b/flink-connectors/flink-connector-nifi/pom.xml
@@ -37,7 +37,7 @@ under the License.
<!-- Allow users to pass custom connector versions -->
<properties>
- <nifi.version>0.6.1</nifi.version>
+ <nifi.version>1.6.0</nifi.version>
</properties>
<dependencies>