You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2017/08/09 13:41:56 UTC
Re: Using Hadoop 2.8.0 in Flink Project for S3A Path Style Access
So you're saying that this works if you manually compile Flink for Hadoop 2.8.0? If yes, I think the solution is that we have to provide binaries for Hadoop 2.8.0. If we did that with a possible Flink 1.3.3 release and starting from Flink 1.4.0, would this be an option for you?
Best,
Aljoscha
> On 11. Jul 2017, at 10:47, Mustafa AKIN <mu...@gmail.com> wrote:
>
> Hi all,
>
> I am trying to use S3 backend with custom endpoint. However, it is not supported in hadoop-aws@2.7.3, I need to use at least 2.8.0 version. The underyling reason is that the requests are being sent as following
>
> DEBUG [main] (AmazonHttpClient.java:337) - Sending Request: HEAD http://mustafa.localhost:9000 <http://mustafa.localhost:9000/> / Headers:
>
> Because "fs.s3a.path.style.access" is not recognized in old version.I want the domain to remain same, the bucket name to be appended in the path (http://localhost:9000/mustafa/.. <http://localhost:9000/>.)
>
> I cannot blindly increase aws-java-sdk version to latest, it causes:
>
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.ClientConfiguration
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:182)
>
> So, If I increase the hadoop-aws to 2.8.0 with latest client, it causes the following error:
>
>
> According to, I need hadoop-aws@2.7.2 and
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#provide-s3-filesystem-dependency <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#provide-s3-filesystem-dependency>
>
> Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation
> at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:194)
>
>
> Should I be excluding hadoop-common from Flink somehow? Building flink from source with mvn clean install -DskipTests -Dhadoop.version=2.8.0 works but I want to manage it via maven as much as possible.
Re: Using Hadoop 2.8.0 in Flink Project for S3A Path Style Access
Posted by Aljoscha Krettek <al...@apache.org>.
I created an issue for this: https://issues.apache.org/jira/browse/FLINK-7413 <https://issues.apache.org/jira/browse/FLINK-7413>
> On 10. Aug 2017, at 16:05, Mustafa AKIN <mu...@gmail.com> wrote:
>
> Yes, it would probably work. I cloned master repo and compiled with 2.8.0 and it worked. It would be nice to have 2.8 binaries since Hadoop 2.8.1 is also released
>
> Mustafa Akın
> www.mustafaak.in <http://www.mustafaak.in/>
>
> On Wed, Aug 9, 2017 at 9:00 PM, Eron Wright <eronwright@gmail.com <ma...@gmail.com>> wrote:
> For reference: [FLINK-6466] Build Hadoop 2.8.0 convenience binaries
>
> On Wed, Aug 9, 2017 at 6:41 AM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
> So you're saying that this works if you manually compile Flink for Hadoop 2.8.0? If yes, I think the solution is that we have to provide binaries for Hadoop 2.8.0. If we did that with a possible Flink 1.3.3 release and starting from Flink 1.4.0, would this be an option for you?
>
> Best,
> Aljoscha
>
>> On 11. Jul 2017, at 10:47, Mustafa AKIN <mustafa91@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi all,
>>
>> I am trying to use S3 backend with custom endpoint. However, it is not supported in hadoop-aws@2.7.3, I need to use at least 2.8.0 version. The underyling reason is that the requests are being sent as following
>>
>> DEBUG [main] (AmazonHttpClient.java:337) - Sending Request: HEAD http://mustafa.localhost:9000 <http://mustafa.localhost:9000/> / Headers:
>>
>> Because "fs.s3a.path.style.access" is not recognized in old version.I want the domain to remain same, the bucket name to be appended in the path (http://localhost:9000/mustafa/.. <http://localhost:9000/>.)
>>
>> I cannot blindly increase aws-java-sdk version to latest, it causes:
>>
>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.ClientConfiguration
>> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:182)
>>
>> So, If I increase the hadoop-aws to 2.8.0 with latest client, it causes the following error:
>>
>>
>> According to, I need hadoop-aws@2.7.2 and
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#provide-s3-filesystem-dependency <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#provide-s3-filesystem-dependency>
>>
>> Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation
>> at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:194)
>>
>>
>> Should I be excluding hadoop-common from Flink somehow? Building flink from source with mvn clean install -DskipTests -Dhadoop.version=2.8.0 works but I want to manage it via maven as much as possible.
>
>
>
Re: Using Hadoop 2.8.0 in Flink Project for S3A Path Style Access
Posted by Mustafa AKIN <mu...@gmail.com>.
Yes, it would probably work. I cloned master repo and compiled with 2.8.0
and it worked. It would be nice to have 2.8 binaries since Hadoop 2.8.1 is
also released
Mustafa Akın
www.mustafaak.in
On Wed, Aug 9, 2017 at 9:00 PM, Eron Wright <er...@gmail.com> wrote:
> For reference: [FLINK-6466] Build Hadoop 2.8.0 convenience binaries
>
> On Wed, Aug 9, 2017 at 6:41 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> So you're saying that this works if you manually compile Flink for Hadoop
>> 2.8.0? If yes, I think the solution is that we have to provide binaries for
>> Hadoop 2.8.0. If we did that with a possible Flink 1.3.3 release and
>> starting from Flink 1.4.0, would this be an option for you?
>>
>> Best,
>> Aljoscha
>>
>> On 11. Jul 2017, at 10:47, Mustafa AKIN <mu...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I am trying to use S3 backend with custom endpoint. However, it is not
>> supported in hadoop-aws@2.7.3, I need to use at least 2.8.0 version. The
>> underyling reason is that the requests are being sent as following
>>
>> DEBUG [main] (AmazonHttpClient.java:337) - Sending Request: HEAD
>> http://mustafa.localhost:9000 / Headers:
>>
>> Because "fs.s3a.path.style.access" is not recognized in old version.I
>> want the domain to remain same, the bucket name to be appended in the path (
>> http://localhost:9000/mustafa/.. <http://localhost:9000/>.)
>>
>> I cannot blindly increase aws-java-sdk version to latest, it causes:
>>
>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
>> com.amazonaws.ClientConfiguration
>> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSys
>> tem.java:182)
>>
>> So, If I increase the hadoop-aws to 2.8.0 with latest client, it causes
>> the following error:
>>
>>
>> According to, I need hadoop-aws@2.7.2 and
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> setup/aws.html#provide-s3-filesystem-dependency
>>
>> Caused by: java.lang.IllegalAccessError: tried to access method
>> org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lor
>> g/apache/hadoop/metrics2/MetricsInfo;J)V from class
>> org.apache.hadoop.fs.s3a.S3AInstrumentation
>> at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3
>> AInstrumentation.java:194)
>>
>>
>> Should I be excluding hadoop-common from Flink somehow? Building flink
>> from source with mvn clean install -DskipTests -Dhadoop.version=2.8.0 works
>> but I want to manage it via maven as much as possible.
>>
>>
>>
>
Re: Using Hadoop 2.8.0 in Flink Project for S3A Path Style Access
Posted by Eron Wright <er...@gmail.com>.
For reference: [FLINK-6466] Build Hadoop 2.8.0 convenience binaries
On Wed, Aug 9, 2017 at 6:41 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> So you're saying that this works if you manually compile Flink for Hadoop
> 2.8.0? If yes, I think the solution is that we have to provide binaries for
> Hadoop 2.8.0. If we did that with a possible Flink 1.3.3 release and
> starting from Flink 1.4.0, would this be an option for you?
>
> Best,
> Aljoscha
>
> On 11. Jul 2017, at 10:47, Mustafa AKIN <mu...@gmail.com> wrote:
>
> Hi all,
>
> I am trying to use S3 backend with custom endpoint. However, it is not
> supported in hadoop-aws@2.7.3, I need to use at least 2.8.0 version. The
> underyling reason is that the requests are being sent as following
>
> DEBUG [main] (AmazonHttpClient.java:337) - Sending Request: HEAD
> http://mustafa.localhost:9000 / Headers:
>
> Because "fs.s3a.path.style.access" is not recognized in old version.I want
> the domain to remain same, the bucket name to be appended in the path (
> http://localhost:9000/mustafa/.. <http://localhost:9000/>.)
>
> I cannot blindly increase aws-java-sdk version to latest, it causes:
>
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> com.amazonaws.ClientConfiguration
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:182)
>
> So, If I increase the hadoop-aws to 2.8.0 with latest client, it causes
> the following error:
>
>
> According to, I need hadoop-aws@2.7.2 and
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/setup/aws.html#provide-s3-filesystem-dependency
>
> Caused by: java.lang.IllegalAccessError: tried to access method
> org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(
> Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class
> org.apache.hadoop.fs.s3a.S3AInstrumentation
> at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(
> S3AInstrumentation.java:194)
>
>
> Should I be excluding hadoop-common from Flink somehow? Building flink
> from source with mvn clean install -DskipTests -Dhadoop.version=2.8.0 works
> but I want to manage it via maven as much as possible.
>
>
>