Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[peterpanne <er...@peukert.org>]

I am facing the same issue in a different setting:
I have a webapplication that calls the local flink execution.
Unfortunately the change in the bootloader chain introduces the described
problem

Boot classloader
App class-loader
webapp classloader (knows the flink-classes)

it seems that the InstantiationUtil from flink replaces the current context
class loader which knows about the flink classes with the blob-classloader
with does not have the webappclassloader as parent:

Boot classloader
App classloader
Flink's Blob classloader

This is obviously the same problem that also appears when running the local
flink election from within an eclipse plugin.

Could you help me here? What should I do to tell flink not to ignore the
webapp-classloader?

Best,
eric






--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9649.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[Stephan Ewen <se...@apache.org>]

I agree, this would be nice to support in Flink.

The important parts are on the client side (which may be embedded). The
classloaders are used there as part of the de-serialization of messages
that contain user-defined types (such as in collect() or in accumulators
and in exception reporting).

How are you starting the program in your web-app, by the way?

If you would be up for contributing this fix, that would be great! One
could approach this in two ways:

  - Simple, but not sure if completely future proof: we take always the
context classloader as the parent, and if that is null, the system class
loader

  - More elaborate: We allow to set a root classloader on the
ExecutionEnvironment. That one should be passed as parent class loader to
the Remote Executor and the Client.


Greetings,
Stephan


On Mon, Dec 21, 2015 at 4:41 PM, peterpanne <er...@peukert.org> wrote:

> I identified the root-cause of the problem:
>
> When the FlinkUserCodeClassLoader ist created it simply instantiates the
> URLClassLoader which takes the System-Classloader as parent. However in the
> web-application-setting the current thread-webapp-classloader should be
> taken. I tried to change the class slightly to take the "right" threads
> class loader as parent.
>  private static class FlinkUserCodeClassLoader extends URLClassLoader {
>                 public FlinkUserCodeClassLoader(URL[] urls) {
>                         super(urls,
> Thread.currentThread().getContextClassLoader());}}
>
> Unfortunately there are multiple places where the system-classloader ist
> taken instead of the thread class loader:
>
> flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
> That solved my problem for the moment.
> However, I am not sure if that introduces problems somewhere else. In
> theory
> it should not be critical since the web-appclassloader simply introduces an
> additional level of classes to look for.
>
> It would cool if that problem could be resolved in flink ;-)
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9653.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: ClassNotFoundException : org.apache.flink.api.common.operators.util.UserCodeObjectWrapper, while trying to run locally

[peterpanne <er...@peukert.org>]

I identified the root-cause of the problem:

When the FlinkUserCodeClassLoader ist created it simply instantiates the
URLClassLoader which takes the System-Classloader as parent. However in the
web-application-setting the current thread-webapp-classloader should be
taken. I tried to change the class slightly to take the "right" threads
class loader as parent.
 private static class FlinkUserCodeClassLoader extends URLClassLoader {
                public FlinkUserCodeClassLoader(URL[] urls) {
                        super(urls,
Thread.currentThread().getContextClassLoader());}}

Unfortunately there are multiple places where the system-classloader ist
taken instead of the thread class loader: 
flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java 
That solved my problem for the moment.
However, I am not sure if that introduces problems somewhere else. In theory
it should not be critical since the web-appclassloader simply introduces an
additional level of classes to look for. 

It would cool if that problem could be resolved in flink ;-)





--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ClassNotFoundException-org-apache-flink-api-common-operators-util-UserCodeObjectWrapper-while-tryingy-tp5922p9653.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.