NoClassDefFoundError with ElasticsearchSink on Yarn

["Hausmann, Steffen" <sh...@amazon.de>]

Hi there,

I\u2019m running a flink program that reads from a Kinesis stream and 
eventually writes to an Elasticsearch2 sink. When I\u2019m running the 
program locally from the IDE, everything seems to work fine, but when 
I\u2019m executing the same program on an EMR cluster with Yarn, a 
NoClassDefFoundError occurs:

java.lang.NoSuchMethodError: 
com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;

        at 
org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)

        at 
org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)

        at 
org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.open(ElasticsearchSink.java:164)

        at 
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)

        at 
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)

        at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:256)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)

        at java.lang.Thread.run(Thread.java:745)

I\u2019ve installed flink on an EMR cluster from the binary distribution 
flink-1.1.1-bin-hadoop27-scala_2.10.tgz. I build the jar file locally 
with mvn clean package(I\u2019ve attached the pom.xml for reference) and copy 
it to the cluster for execution. There is a thread on this list that 
seems to be related, but I\u2019m afraid I couldn\u2019t draw any conclusions from it:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/classpath-issue-on-yarn-tt6442.html#none


Any idea, what\u2019s wrong?

Thanks,

Steffen

Re: NoClassDefFoundError with ElasticsearchSink on Yarn

[Fabian Hueske <fh...@gmail.com>]

Hi Steffen,

this looks like a Guava version mismatch to me.
Are you running exactly the same program on your local machine or did you
add dependencies to run it on the cluster (e.g. Kinesis).
Maybe Kinesis and Elasticsearch are using different Guava versions?

Best, Fabian

2016-09-01 10:45 GMT+02:00 Hausmann, Steffen <sh...@amazon.de>:

> Hi there,
>
> I’m running a flink program that reads from a Kinesis stream and
> eventually writes to an Elasticsearch2 sink. When I’m running the program
> locally from the IDE, everything seems to work fine, but when I’m executing
> the same program on an EMR cluster with Yarn, a NoClassDefFoundError occurs:
>
> java.lang.NoSuchMethodError: com.google.common.util.concurr
> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>
>        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.
> java:190)
>
>        at org.elasticsearch.client.transport.TransportClient$Builder.
> build(TransportClient.java:133)
>
>        at org.apache.flink.streaming.connectors.elasticsearch2.Elastic
> searchSink.open(ElasticsearchSink.java:164)
>
>        at org.apache.flink.api.common.functions.util.FunctionUtils.ope
> nFunction(FunctionUtils.java:38)
>
>        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOp
> erator.open(AbstractUdfStreamOperator.java:91)
>
>        at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO
> perators(StreamTask.java:376)
>
>        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
> StreamTask.java:256)
>
>        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>
>        at java.lang.Thread.run(Thread.java:745)
>
> I’ve installed flink on an EMR cluster from the binary distribution
> flink-1.1.1-bin-hadoop27-scala_2.10.tgz. I build the jar file locally
> with mvn clean package(I’ve attached the pom.xml for reference) and copy it
> to the cluster for execution. There is a thread on this list that seems to
> be related, but I’m afraid I couldn’t draw any conclusions from it:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.
> nabble.com/classpath-issue-on-yarn-tt6442.html#none
>
>
> Any idea, what’s wrong?
>
> Thanks,
>
> Steffen
>
>