[jira] [Commented] (FLINK-5691) Creating Reporter for elasticsearch 5.1.X causing conflicts of io.netty library

["Chesnay Schepler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878299#comment-15878299 ] 

Chesnay Schepler commented on FLINK-5691:
-----------------------------------------

There aren't to many options here; you can try downgrading the netty dependency of ES of try to shade all ES dependencies in your reporter jar.

I'm uncertain if we can shade netty away in Flink, maybe [~uce] knows more about this.

> Creating Reporter for elasticsearch 5.1.X causing conflicts of io.netty library
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-5691
>                 URL: https://issues.apache.org/jira/browse/FLINK-5691
>             Project: Flink
>          Issue Type: Bug
>          Components: Metrics
>    Affects Versions: 1.3.0
>         Environment: linux 
>            Reporter: prabhat kumar
>              Labels: features, maven
>
> Trying to write reporter for elasticsearch 5.1.x using TransportClient which internally using io.netty version 4.1.6 which has a call to a method which is not present in flink io.netty verion 4.0.27 causing error
> ***********************************************************************************
> Exception in thread "elasticsearch[_client_][management][T#1]" java.lang.NoSuchMethodError: io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;
>         at org.elasticsearch.transport.netty4.Netty4Utils.toByteBuf(Netty4Utils.java:78)
>         at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:449)
>         at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:91)
>         at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:976)
>         at org.elasticsearch.transport.TcpTransport.sendRequest(TcpTransport.java:958)
>         at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:520)
>         at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:465)
>         at org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1.run(TransportClientNodesService.java:482)
>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> ***********************************************************************************
> trying to upgrade the jar but in pom.xml of flink it's mentioned not to upgrade else there could be memory issue 
> <dependency>
> 				<groupId>io.netty</groupId>
> 				<artifactId>netty-all</artifactId>
> 				<!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> 				contain an improvement by Netty, which slices a Netty buffer
> 				instead of doing a memory copy [1] in the
> 				LengthFieldBasedFrameDecoder. In some situations, this
> 				interacts badly with our Netty pipeline leading to OutOfMemory
> 				errors.
> 				[1] https://github.com/netty/netty/issues/3704 -->
> 				<version>4.0.27.Final</version>
> 			</dependency>
> Please suggest a workaround.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)