Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[gkrastev <jo...@gmail.com>]

Hi,I'm facing the exact same issue when running a custom Flink interpreter in
Zeppelin, so it's a case of special classloader. I'm adding the Flink jars
to the Scala REPL's classpath manually, so I end up with the following
classloader chain:1. Boot classloader2. App classloader3. Scala classloader
-> has Flink on the classpath4. Translating classloader for the Scala REPLSo
the class is accessible in the vanilla Zeppelin shell. E.g. the following
code works fine:
println(classOf[org.apache.flink.api.common.operators.util.UserCodeObjectWrapper[String]])
However, the classloader chain in the Task Manager looks differently:1. Boot
classloader2. App classloader3. Flink's Blob classloaderAnd the following
code fails:
import org.apache.flink.api.scala._env.fromCollection(1 to 100).reduce { _ +
_ }.collect()
because the Flink jars are no longer on the classpath. The question is, why
doesn't Flink pickup the current classloader/classpath when starting Task
Managers, and can I add entries to their classpath manually?
Robert Metzger wrote
> Can you give us some context what you are trying to do?It sounds a lot
> like you would like to develop a Flink Plugin for Eclipse?And that plugin
> needs Flink dependencies?Sorry for asking so many questions, but the issue
> you are facing sounds alot like a) a very special classloader or b)
> corrupt files.I think some more info on what you are doing might help us
> to understandthe issue to give you better help.





--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p7870.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[Stephan Ewen <se...@apache.org>]

I agree, this would be nice to support in Flink.

The important parts are on the client side (which may be embedded). The
classloaders are used there as part of the de-serialization of messages
that contain user-defined types (such as in collect() or in accumulators
and in exception reporting).

How are you starting the program in your web-app, by the way?

If you would be up for contributing this fix, that would be great! One
could approach this in two ways:

  - Simple, but not sure if completely future proof: we take always the
context classloader as the parent, and if that is null, the system class
loader

  - More elaborate: We allow to set a root classloader on the
ExecutionEnvironment. That one should be passed as parent class loader to
the Remote Executor and the Client.


Greetings,
Stephan


On Mon, Dec 21, 2015 at 4:41 PM, peterpanne <er...@peukert.org> wrote:

> I identified the root-cause of the problem:
>
> When the FlinkUserCodeClassLoader ist created it simply instantiates the
> URLClassLoader which takes the System-Classloader as parent. However in the
> web-application-setting the current thread-webapp-classloader should be
> taken. I tried to change the class slightly to take the "right" threads
> class loader as parent.
>  private static class FlinkUserCodeClassLoader extends URLClassLoader {
>                 public FlinkUserCodeClassLoader(URL[] urls) {
>                         super(urls,
> Thread.currentThread().getContextClassLoader());}}
>
> Unfortunately there are multiple places where the system-classloader ist
> taken instead of the thread class loader:
>
> flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
> That solved my problem for the moment.
> However, I am not sure if that introduces problems somewhere else. In
> theory
> it should not be critical since the web-appclassloader simply introduces an
> additional level of classes to look for.
>
> It would cool if that problem could be resolved in flink ;-)
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9653.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[peterpanne <er...@peukert.org>]

I identified the root-cause of the problem:

When the FlinkUserCodeClassLoader ist created it simply instantiates the
URLClassLoader which takes the System-Classloader as parent. However in the
web-application-setting the current thread-webapp-classloader should be
taken. I tried to change the class slightly to take the "right" threads
class loader as parent.
 private static class FlinkUserCodeClassLoader extends URLClassLoader {
                public FlinkUserCodeClassLoader(URL[] urls) {
                        super(urls,
Thread.currentThread().getContextClassLoader());}}

Unfortunately there are multiple places where the system-classloader ist
taken instead of the thread class loader: 
flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java 
That solved my problem for the moment.
However, I am not sure if that introduces problems somewhere else. In theory
it should not be critical since the web-appclassloader simply introduces an
additional level of classes to look for. 

It would cool if that problem could be resolved in flink ;-)





--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9653.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[peterpanne <er...@peukert.org>]

I am facing the same issue in a different setting:
I have a webapplication that calls the local flink execution.
Unfortunately the change in the bootloader chain introduces the described
problem

Boot classloader
App class-loader
webapp classloader (knows the flink-classes)

it seems that the InstantiationUtil from flink replaces the current context
class loader which knows about the flink classes with the blob-classloader
with does not have the webappclassloader as parent:

Boot classloader
App classloader
Flink's Blob classloader

This is obviously the same problem that also appears when running the local
flink election from within an eclipse plugin.

Could you help me here? What should I do to tell flink not to ignore the
webapp-classloader?

Best,
eric






--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9649.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[Stephan Ewen <se...@apache.org>]

Effectively, the method with the blob-classloader / parent classloader is
exactly intended to do that: Start with the app class path and add some
jars in addition, if you need to.

On Mon, Sep 7, 2015 at 1:05 PM, Stephan Ewen <se...@apache.org> wrote:

> The BlobClassloader uses the App classloader as the parent. That should
> make the classloading go first to the app classpath (TaskManager's
> classpath) and then to the user-defined code.
> Is that broken somehow?
>
>
>
> On Sun, Sep 6, 2015 at 1:40 PM, gkrastev <jo...@gmail.com> wrote:
>
>> Hi,I'm facing the exact same issue when running a custom Flink
>> interpreter in
>> Zeppelin, so it's a case of special classloader. I'm adding the Flink jars
>> to the Scala REPL's classpath manually, so I end up with the following
>> classloader chain:1. Boot classloader2. App classloader3. Scala
>> classloader
>> -> has Flink on the classpath4. Translating classloader for the Scala
>> REPLSo
>> the class is accessible in the vanilla Zeppelin shell. E.g. the following
>> code works fine:
>>
>> println(classOf[org.apache.flink.api.common.operators.util.UserCodeObjectWrapper[String]])
>> However, the classloader chain in the Task Manager looks differently:1.
>> Boot
>> classloader2. App classloader3. Flink's Blob classloaderAnd the following
>> code fails:
>> import org.apache.flink.api.scala._env.fromCollection(1 to 100).reduce {
>> _ +
>> _ }.collect()
>> because the Flink jars are no longer on the classpath. The question is,
>> why
>> doesn't Flink pickup the current classloader/classpath when starting Task
>> Managers, and can I add entries to their classpath manually?
>> Robert Metzger wrote
>> > Can you give us some context what you are trying to do?It sounds a lot
>> > like you would like to develop a Flink Plugin for Eclipse?And that
>> plugin
>> > needs Flink dependencies?Sorry for asking so many questions, but the
>> issue
>> > you are facing sounds alot like a) a very special classloader or b)
>> > corrupt files.I think some more info on what you are doing might help us
>> > to understandthe issue to give you better help.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p7870.html
>> Sent from the Apache Flink Mailing List archive. mailing list archive at
>> Nabble.com.
>>
>
>

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[Stephan Ewen <se...@apache.org>]

The BlobClassloader uses the App classloader as the parent. That should
make the classloading go first to the app classpath (TaskManager's
classpath) and then to the user-defined code.
Is that broken somehow?



On Sun, Sep 6, 2015 at 1:40 PM, gkrastev <jo...@gmail.com> wrote:

> Hi,I'm facing the exact same issue when running a custom Flink interpreter
> in
> Zeppelin, so it's a case of special classloader. I'm adding the Flink jars
> to the Scala REPL's classpath manually, so I end up with the following
> classloader chain:1. Boot classloader2. App classloader3. Scala classloader
> -> has Flink on the classpath4. Translating classloader for the Scala
> REPLSo
> the class is accessible in the vanilla Zeppelin shell. E.g. the following
> code works fine:
>
> println(classOf[org.apache.flink.api.common.operators.util.UserCodeObjectWrapper[String]])
> However, the classloader chain in the Task Manager looks differently:1.
> Boot
> classloader2. App classloader3. Flink's Blob classloaderAnd the following
> code fails:
> import org.apache.flink.api.scala._env.fromCollection(1 to 100).reduce { _
> +
> _ }.collect()
> because the Flink jars are no longer on the classpath. The question is, why
> doesn't Flink pickup the current classloader/classpath when starting Task
> Managers, and can I add entries to their classpath manually?
> Robert Metzger wrote
> > Can you give us some context what you are trying to do?It sounds a lot
> > like you would like to develop a Flink Plugin for Eclipse?And that plugin
> > needs Flink dependencies?Sorry for asking so many questions, but the
> issue
> > you are facing sounds alot like a) a very special classloader or b)
> > corrupt files.I think some more info on what you are doing might help us
> > to understandthe issue to give you better help.
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p7870.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>