You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by nragon <nu...@wedotechnologies.com> on 2017/07/26 10:53:14 UTC
Logback user class
Hi,
I executing the following snippet on two different environments.
--------
StreamExecutionEnvironment streamEnv =
StreamExecutionEnvironment.createRemoteEnvironment("xxxxxxxxx", 6123);
streamEnv.setParallelism(10);
StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(streamEnv);
DataStream<Tuple2<Integer, String>> stream1 =
streamEnv.fromElements(new Tuple2<>(1, "hello"));
tableEnv.registerDataStream("a", stream1);
DataStream<Row> dataStream =
tableEnv.toAppendStream(tableEnv.sql("SELECT f0, f1 from a"), Row.class);
dataStream.print();
streamEnv.execute("job");
--------
When executing through IDE everything works fine. However if I execute the
same code loaded by a different class loader used on our application I get
the following error, on flink side.
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load
user class: ch.qos.logback.classic.Logger
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:229)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:95)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:230)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
On the client side we are using logback but according with the snippet there
is no need to load logback on flink side. Is there any reference binding
while building a stream graph or job that might reference logback as
dependency? Or does flink assumes client log mechanism?
Thanks
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Logback user class
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
It seems there is a bug in the internal Table API operators: they store the Logger that is available on the Client machine and deserialising that Logger fails on the cluster. I created a Jira issue for this: https://issues.apache.org/jira/browse/FLINK-7398 <https://issues.apache.org/jira/browse/FLINK-7398>
Best,
Aljoscha
> On 26. Jul 2017, at 15:54, Nuno Rafael Goncalves <Nu...@wedotechnologies.com> wrote:
>
> Even though I have executed the same code with intelliJ and works fine? <>
> The only difference is that intelliJ is using log4j and the other application is using logback.
> Moreover, the snippet is quite simple, it does not reference any user class other than flink's
> Does flink uses user loaded log implementation and tries to use it on server side? Which in this case would justify the logback dependency.
>
> From: Stephan Ewen [mailto:sewen@apache.org <ma...@apache.org>]
> Sent: 26 de julho de 2017 14:42
> To: Nuno Rafael Goncalves <Nuno.Goncalves@wedotechnologies.com <ma...@wedotechnologies.com>>
> Subject: Re: Logback user class
>
> You seem to have a reference to the Logback Logger somewhere in your code.
>
> The class for that logger seems to be not in the user code jar file, or in the flink lib directory.
> Since Flink does not bundle logback by itself, you need to package this dependency explicitly or add the logback jar to the lib folder.
>
> On Wed, Jul 26, 2017 at 3:01 PM, nragon <nuno.goncalves@wedotechnologies.com <ma...@wedotechnologies.com>> wrote:
> I've changed that line and compiled it into lib/. Error remains.
> I'm running a local custer with start-local.sh
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464p14469.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464p14469.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
RE: Logback user class
Posted by Nuno Rafael Goncalves <Nu...@wedotechnologies.com>.
Even though I have executed the same code with intelliJ and works fine?
The only difference is that intelliJ is using log4j and the other application is using logback.
Moreover, the snippet is quite simple, it does not reference any user class other than flink's
Does flink uses user loaded log implementation and tries to use it on server side? Which in this case would justify the logback dependency.
From: Stephan Ewen [mailto:sewen@apache.org]
Sent: 26 de julho de 2017 14:42
To: Nuno Rafael Goncalves <Nu...@wedotechnologies.com>
Subject: Re: Logback user class
You seem to have a reference to the Logback Logger somewhere in your code.
The class for that logger seems to be not in the user code jar file, or in the flink lib directory.
Since Flink does not bundle logback by itself, you need to package this dependency explicitly or add the logback jar to the lib folder.
On Wed, Jul 26, 2017 at 3:01 PM, nragon <nu...@wedotechnologies.com>> wrote:
I've changed that line and compiled it into lib/. Error remains.
I'm running a local custer with start-local.sh
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464p14469.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Logback user class
Posted by nragon <nu...@wedotechnologies.com>.
I've changed that line and compiled it into lib/. Error remains.
I'm running a local custer with start-local.sh
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464p14469.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Logback user class
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at FLINK-6767.
On Wed, Jul 26, 2017 at 3:53 AM, nragon <nuno.goncalves@wedotechnologies.com
> wrote:
> Hi,
>
> I executing the following snippet on two different environments.
>
> --------
> StreamExecutionEnvironment streamEnv =
> StreamExecutionEnvironment.createRemoteEnvironment("xxxxxxxxx", 6123);
> streamEnv.setParallelism(10);
> StreamTableEnvironment tableEnv =
> TableEnvironment.getTableEnvironment(streamEnv);
> DataStream<Tuple2<Integer, String>> stream1 =
> streamEnv.fromElements(new Tuple2<>(1, "hello"));
> tableEnv.registerDataStream("a", stream1);
> DataStream<Row> dataStream =
> tableEnv.toAppendStream(tableEnv.sql("SELECT f0, f1 from a"), Row.class);
> dataStream.print();
> streamEnv.execute("job");
> --------
>
> When executing through IDE everything works fine. However if I execute the
> same code loaded by a different class loader used on our application I get
> the following error, on flink side.
>
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load
> user class: ch.qos.logback.classic.Logger
> ClassLoader info: URL ClassLoader:
> Class not resolvable through given classloader.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(
> StreamConfig.java:229)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.<
> init>(OperatorChain.java:95)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:230)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:745)
>
> On the client side we are using logback but according with the snippet
> there
> is no need to load logback on flink side. Is there any reference binding
> while building a stream graph or job that might reference logback as
> dependency? Or does flink assumes client log mechanism?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Logback-user-class-tp14464.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>