You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Satish Saley <sa...@apache.org> on 2020/08/14 18:25:42 UTC
Flink S3 Hadoop dependencies
Hi team,
Was there a reason for not shading hadoop-common
https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44
? This is leaking lots of classes such as guava and causing issues in our
flink application.
I see that hadoop-common classes were shaded in earlier versions
https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0
Stacktrace :
Caused by: java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
at
io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
at
io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
at
io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)
Re: Flink S3 Hadoop dependencies
Posted by Chesnay Schepler <ch...@apache.org>.
Filesystems are supposed to be used as plugins (by putting the jars
under plugins/ instead of lib/), in which case they are loaded
separately from other classes, specifically user-code.
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/plugins.html
On 14/08/2020 20:25, Satish Saley wrote:
> Hi team,
>
> Was there a reason for not shading hadoop-common
> https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44
> ? This is leaking lots of classes such as guava and causing issues in
> our flink application.
> I see that hadoop-common classes were shaded in earlier versions
> https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0
>
> Stacktrace :
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
> at io.grpc.Metadata$Key.validateName(Metadata.java:742)
> at io.grpc.Metadata$Key.<init>(Metadata.java:750)
> at io.grpc.Metadata$Key.<init>(Metadata.java:668)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
> at io.grpc.Metadata$Key.of(Metadata.java:705)
> at io.grpc.Metadata$Key.of(Metadata.java:701)
> at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
> at
> io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
> at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)
Re: Flink S3 Hadoop dependencies
Posted by Chesnay Schepler <ch...@apache.org>.
Filesystems are supposed to be used as plugins (by putting the jars
under plugins/ instead of lib/), in which case they are loaded
separately from other classes, specifically user-code.
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/plugins.html
On 14/08/2020 20:25, Satish Saley wrote:
> Hi team,
>
> Was there a reason for not shading hadoop-common
> https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44
> ? This is leaking lots of classes such as guava and causing issues in
> our flink application.
> I see that hadoop-common classes were shaded in earlier versions
> https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0
>
> Stacktrace :
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
> at io.grpc.Metadata$Key.validateName(Metadata.java:742)
> at io.grpc.Metadata$Key.<init>(Metadata.java:750)
> at io.grpc.Metadata$Key.<init>(Metadata.java:668)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
> at io.grpc.Metadata$Key.of(Metadata.java:705)
> at io.grpc.Metadata$Key.of(Metadata.java:701)
> at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
> at
> io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
> at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)