You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ch...@apache.org on 2019/08/01 10:15:52 UTC
[flink] branch s3_test updated (7d0d032 -> 6d13434)
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a change to branch s3_test
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 7d0d032 test configured s3 provider
new 8e26246 FLINK-13044 - Fix for wrong shading of AWS SDK in flink-s3-fs-hadoop
new 6d13434 Fix unit tests
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
flink-filesystems/flink-s3-fs-hadoop/pom.xml | 4 ++++
.../main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java | 4 ++--
.../java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java | 2 +-
3 files changed, 7 insertions(+), 3 deletions(-)
[flink] 01/02: FLINK-13044 - Fix for wrong shading of AWS SDK in
flink-s3-fs-hadoop
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch s3_test
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 8e2624662bc7d0d4585035339dd0dd92bc9718aa
Author: Sebastian Just <Se...@comcast.com>
AuthorDate: Mon Jul 1 09:07:27 2019 -0400
FLINK-13044 - Fix for wrong shading of AWS SDK in flink-s3-fs-hadoop
Due to the bug MSHADE-156 in Maven's shading plugin [1] the merging
of the AWS SDK also causes string literals to be changed in Flink
classes.
This commit follows the workaround shown in [1] to workaround the
shading issue.
Also fixing the wrong package name in the Flink code for the
shaded AWS SDK classes.
[1] https://issues.apache.org/jira/browse/MSHADE-156
---
flink-filesystems/flink-s3-fs-hadoop/pom.xml | 4 ++++
.../main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java | 4 ++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/flink-filesystems/flink-s3-fs-hadoop/pom.xml b/flink-filesystems/flink-s3-fs-hadoop/pom.xml
index b828858..5c558d0 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/pom.xml
+++ b/flink-filesystems/flink-s3-fs-hadoop/pom.xml
@@ -111,6 +111,10 @@ under the License.
<pattern>org.apache.flink.runtime.util</pattern>
<shadedPattern>org.apache.flink.fs.s3hadoop.common</shadedPattern>
</relocation>
+ <relocation>
+ <pattern>com.UNSHADE.amazon</pattern>
+ <shadedPattern>com.amazon</shadedPattern>
+ </relocation>
</relocations>
<filters>
<filter>
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java b/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
index 6cad051..c3e8219 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
@@ -41,11 +41,11 @@ public class S3FileSystemFactory extends AbstractS3FileSystemFactory {
private static final Logger LOG = LoggerFactory.getLogger(S3FileSystemFactory.class);
- private static final Set<String> PACKAGE_PREFIXES_TO_SHADE = Collections.singleton("com.amazonaws.");
+ private static final Set<String> PACKAGE_PREFIXES_TO_SHADE = Collections.singleton("com.UNSHADE.amazonaws.");
private static final Set<String> CONFIG_KEYS_TO_SHADE = Collections.singleton("fs.s3a.aws.credentials.provider");
- private static final String FLINK_SHADING_PREFIX = "org.apache.flink.fs.s3hadoop.shaded.";
+ private static final String FLINK_SHADING_PREFIX = "org.apache.flink.fs.s3base.shaded.";
private static final String[] FLINK_CONFIG_PREFIXES = { "s3.", "s3a.", "fs.s3a." };
[flink] 02/02: Fix unit tests
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch s3_test
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 6d13434688e879f7c5ca79fe00c004d0205f7329
Author: Sebastian Just <Se...@comcast.com>
AuthorDate: Thu Jul 11 10:23:49 2019 -0400
Fix unit tests
---
.../test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java
index 57500f3..6e7ec96 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HadoopS3FileSystemTest.java
@@ -39,7 +39,7 @@ public class HadoopS3FileSystemTest {
configLoader.setFlinkConfig(conf);
org.apache.hadoop.conf.Configuration hadoopConfig = configLoader.getOrLoadHadoopConfig();
- assertEquals("org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider",
+ assertEquals("com.amazonaws.auth.ContainerCredentialsProvider",
hadoopConfig.get("fs.s3a.aws.credentials.provider"));
}