FLINK-5236. flink 1.1.3 and modifying the classpath to use another version of fasterxml.

[Estela Gallardo Zapata <eg...@oesia.com>]

Hello everyone,


I'm new in  the flink mailing-list :)  and I'll try to be respectful and helpful.

I'm writing  to this mailing list, due to an issue l have with Flink 1.1.3 and my application.jar, related to this previous thread: https://issues.apache.org/jira/browse/FLINK-5236


I use Intelli-J to run locally my maven application and it runs correctly with flink 1.1.3 and scala 2.10.

But when I try to run my application's jar in the cluster, it seems that there is an issue with library faxterxml.

It seems that Flink uses a different version of this library and, as it overwrites the user-classpath, I can't get my application running in the cluster.

I've tried modifying the Hadoop-conf file, making a flat-jar, modifying HADOOP_USER_CLASSPATH_FIRST, copy my application's jar directly in flink/lib...


Could someone please give me some more clues or help me clarify the explanation in the mailling list?


Thank you very much.

Estela.

Re: FLINK-5236. flink 1.1.3 and modifying the classpath to use another version of fasterxml.

[Till Rohrmann <tr...@apache.org>]

Hi Estela,

it's correct that Flink's runtime has a dependency on

com.fasterxml.jackson.core:jackson-core:jar:2.7.4:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.7.4:compile

First of all it's important to understand where the additional dependency
comes from. Does your application has a dependency on fasterxml or does the
Flink cluster does not start because Hadoop brings in a different version
of fasterxml? In case of the latter, which Hadoop version are you running?
In case of the former, can you change the user code fasterxml version to
2.7.4?

The job manager log under log/flink-xxxx-jobmanager-xxxx.log will print all
dependencies in the classpath of the job manager. This information would
also be interesting.

As a short term solution, you could try to build a custom Flink version
where you adapt the jackson.version in the pom.xml so that it's matching
your fasterxml version. As a midterm solution, it might be necessary to
shade fasterxml in Flink and as a long term solution, we should try to not
expose all system classloader dependencies to the user code class loader.
But that's something Flink has to do.

Cheers,
Till


On Tue, Jan 10, 2017 at 10:10 PM, Estela Gallardo Zapata <
egallardo@oesia.com> wrote:

> Hello everyone,
>
>
> I'm new in  the flink mailing-list :)  and I'll try to be respectful and
> helpful.
>
> I'm writing  to this mailing list, due to an issue l have with Flink 1.1.3
> and my application.jar, related to this previous thread:
> https://issues.apache.org/jira/browse/FLINK-5236
>
>
> I use Intelli-J to run locally my maven application and it runs correctly
> with flink 1.1.3 and scala 2.10.
>
> But when I try to run my application's jar in the cluster, it seems that
> there is an issue with library faxterxml.
>
> It seems that Flink uses a different version of this library and, as it
> overwrites the user-classpath, I can't get my application running in the
> cluster.
>
> I've tried modifying the Hadoop-conf file, making a flat-jar, modifying
> HADOOP_USER_CLASSPATH_FIRST, copy my application's jar directly in
> flink/lib...
>
>
> *Could someone please give me some more clues or help me clarify the
> explanation in the mailling list?*
>
>
> *Thank you very much.*
>
> *Estela.*
>