Class loading and job versioning

[Michal Budzyn <mi...@zalando.de>]

Hi all,
We had a class versioning problem within Flink Job.
The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
The problem was solved by relocating job classes with shade plug-in.

Does flink separate jobs from each other to avoid class conflicts between them and the platform ?
Is job versioning supported or is shading always required ?

Regards,
Michal

Re: Class loading and job versioning

[Stephan Ewen <se...@apache.org>]

Hi!

I agree, that shading is tedious. It seems to be a pretty fundamental Java
problem that exists in all those Java-based frameworks.

The only way I know how to solve this is having fewer dependencies in the
framework code.
Right now, the JVMs that execute the user code have for example a
dependency on Hadoop by default, to use its Filesystem connectors to write
checkpoints to HDFS or S3. That Hadoop dependency comes with many
transitive dependencies by itself...

Greetings,
Stephan


On Wed, Jul 20, 2016 at 5:18 PM, Michal Budzyn <
michal.budzyn.extern@zalando.de> wrote:

> Hi Stephan,
>
> IMO the platform needs better jobs isolation. All this shading shouldn't
> be required at all.
> Michal
>
>
>
>
> On 20.07.2016 16:18, Stephan Ewen wrote:
>
> Hi Michael!
>
> The only safe way in Java to isolate the user code from the platform would
> be to completely run them in different JVMs.
>
> Other than that, Max's method to ensure the correct instantiation should
> work in most cases.
>
> We also continuously try to have fewer dependencies in Flink, to that
> there is less to clash.
>
> Stephan
>
>
> On Wed, Jul 20, 2016 at 3:53 PM, Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Sure. No Problem.
>>
>> The issue is a bit more involved. You're right, the user classes have
>> precedence over the Flink classpath. So your classes were probably
>> loaded fine. However, the user code also calls Flink code which can
>> use a library version different from the job jar library. And boom, it
>> crashes because the job jar library has been loaded previously :)
>>
>> The only way we could circumvent this problem would be to explicitly
>> set a different classloader and load Flink internal classes via
>> reflection. I think the performance penalty for this would be way too
>> high.
>>
>> Best,
>> Max
>>
>> On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
>> <mi...@zalando.de> wrote:
>> > Thanks for the prompt replay.
>> >
>> > You are right. The conflict was between "com.fasterxml.jackson.core"
>> libs.
>> >
>> > I am just wondering. If the the jobs were separted from the platform,
>> >
>> > the jobs libs should have precedence and no versioning problem should
>> have
>> > happened.
>> >
>> > Regards,
>> >
>> > Michal
>> >
>> >
>> >
>> > On 20.07.2016 14:00, Maximilian Michels wrote:
>> >>
>> >> Hi Michal,
>> >>
>> >> I couldn't find Joda in flink-dist. Possibly there is some other clash?
>> >>
>> >> There are two potential issues here:
>> >>
>> >> 1) Flink shades some libraries (Guava) but not all. If you use a
>> >> version of a library in your Flink job which doesn't match the one in
>> >> flink-dist, you're bound for trouble.
>> >>
>> >> 2) Flink separates jobs from each other to avoid potential class
>> >> version mismatches. Each job has its own classloader. In this sense,
>> >> "job versioning" is supported.
>> >>
>> >> Cheers,
>> >> Max
>> >>
>> >> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
>> >> <mi...@zalando.de> wrote:
>> >>>
>> >>> Hi all,
>> >>> We had a class versioning problem within Flink Job.
>> >>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>> >>> The problem was solved by relocating job classes with shade plug-in.
>> >>>
>> >>> Does flink separate jobs from each other to avoid class conflicts
>> between
>> >>> them and the platform ?
>> >>> Is job versioning supported or is shading always required ?
>> >>>
>> >>> Regards,
>> >>> Michal
>> >>>
>> >
>>
>
>
>

Re: Class loading and job versioning

[Michal Budzyn <mi...@zalando.de>]

Hi Stephan,

IMO the platform needs better jobs isolation. All this shading shouldn't 
be required at all.

Michal




On 20.07.2016 16:18, Stephan Ewen wrote:
> Hi Michael!
>
> The only safe way in Java to isolate the user code from the platform 
> would be to completely run them in different JVMs.
>
> Other than that, Max's method to ensure the correct instantiation 
> should work in most cases.
>
> We also continuously try to have fewer dependencies in Flink, to that 
> there is less to clash.
>
> Stephan
>
>
> On Wed, Jul 20, 2016 at 3:53 PM, Maximilian Michels <mxm@apache.org 
> <ma...@apache.org>> wrote:
>
>     Sure. No Problem.
>
>     The issue is a bit more involved. You're right, the user classes have
>     precedence over the Flink classpath. So your classes were probably
>     loaded fine. However, the user code also calls Flink code which can
>     use a library version different from the job jar library. And boom, it
>     crashes because the job jar library has been loaded previously :)
>
>     The only way we could circumvent this problem would be to explicitly
>     set a different classloader and load Flink internal classes via
>     reflection. I think the performance penalty for this would be way too
>     high.
>
>     Best,
>     Max
>
>     On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
>     <michal.budzyn.extern@zalando.de
>     <ma...@zalando.de>> wrote:
>     > Thanks for the prompt replay.
>     >
>     > You are right. The conflict was between
>     "com.fasterxml.jackson.core" libs.
>     >
>     > I am just wondering. If the the jobs were separted from the
>     platform,
>     >
>     > the jobs libs should have precedence and no versioning problem
>     should have
>     > happened.
>     >
>     > Regards,
>     >
>     > Michal
>     >
>     >
>     >
>     > On 20.07.2016 14:00, Maximilian Michels wrote:
>     >>
>     >> Hi Michal,
>     >>
>     >> I couldn't find Joda in flink-dist. Possibly there is some
>     other clash?
>     >>
>     >> There are two potential issues here:
>     >>
>     >> 1) Flink shades some libraries (Guava) but not all. If you use a
>     >> version of a library in your Flink job which doesn't match the
>     one in
>     >> flink-dist, you're bound for trouble.
>     >>
>     >> 2) Flink separates jobs from each other to avoid potential class
>     >> version mismatches. Each job has its own classloader. In this
>     sense,
>     >> "job versioning" is supported.
>     >>
>     >> Cheers,
>     >> Max
>     >>
>     >> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
>     >> <michal.budzyn.extern@zalando.de
>     <ma...@zalando.de>> wrote:
>     >>>
>     >>> Hi all,
>     >>> We had a class versioning problem within Flink Job.
>     >>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>     >>> The problem was solved by relocating job classes with shade
>     plug-in.
>     >>>
>     >>> Does flink separate jobs from each other to avoid class
>     conflicts between
>     >>> them and the platform ?
>     >>> Is job versioning supported or is shading always required ?
>     >>>
>     >>> Regards,
>     >>> Michal
>     >>>
>     >
>
>

Re: Class loading and job versioning

[Michal Budzyn <mi...@zalando.de>]

Hi Stephan,

IMO the platform needs better jobs isolation. All this shading shouldn't 
be required at all.

Michal




On 20.07.2016 16:18, Stephan Ewen wrote:
> Hi Michael!
>
> The only safe way in Java to isolate the user code from the platform 
> would be to completely run them in different JVMs.
>
> Other than that, Max's method to ensure the correct instantiation 
> should work in most cases.
>
> We also continuously try to have fewer dependencies in Flink, to that 
> there is less to clash.
>
> Stephan
>
>
> On Wed, Jul 20, 2016 at 3:53 PM, Maximilian Michels <mxm@apache.org 
> <ma...@apache.org>> wrote:
>
>     Sure. No Problem.
>
>     The issue is a bit more involved. You're right, the user classes have
>     precedence over the Flink classpath. So your classes were probably
>     loaded fine. However, the user code also calls Flink code which can
>     use a library version different from the job jar library. And boom, it
>     crashes because the job jar library has been loaded previously :)
>
>     The only way we could circumvent this problem would be to explicitly
>     set a different classloader and load Flink internal classes via
>     reflection. I think the performance penalty for this would be way too
>     high.
>
>     Best,
>     Max
>
>     On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
>     <michal.budzyn.extern@zalando.de
>     <ma...@zalando.de>> wrote:
>     > Thanks for the prompt replay.
>     >
>     > You are right. The conflict was between
>     "com.fasterxml.jackson.core" libs.
>     >
>     > I am just wondering. If the the jobs were separted from the
>     platform,
>     >
>     > the jobs libs should have precedence and no versioning problem
>     should have
>     > happened.
>     >
>     > Regards,
>     >
>     > Michal
>     >
>     >
>     >
>     > On 20.07.2016 14:00, Maximilian Michels wrote:
>     >>
>     >> Hi Michal,
>     >>
>     >> I couldn't find Joda in flink-dist. Possibly there is some
>     other clash?
>     >>
>     >> There are two potential issues here:
>     >>
>     >> 1) Flink shades some libraries (Guava) but not all. If you use a
>     >> version of a library in your Flink job which doesn't match the
>     one in
>     >> flink-dist, you're bound for trouble.
>     >>
>     >> 2) Flink separates jobs from each other to avoid potential class
>     >> version mismatches. Each job has its own classloader. In this
>     sense,
>     >> "job versioning" is supported.
>     >>
>     >> Cheers,
>     >> Max
>     >>
>     >> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
>     >> <michal.budzyn.extern@zalando.de
>     <ma...@zalando.de>> wrote:
>     >>>
>     >>> Hi all,
>     >>> We had a class versioning problem within Flink Job.
>     >>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>     >>> The problem was solved by relocating job classes with shade
>     plug-in.
>     >>>
>     >>> Does flink separate jobs from each other to avoid class
>     conflicts between
>     >>> them and the platform ?
>     >>> Is job versioning supported or is shading always required ?
>     >>>
>     >>> Regards,
>     >>> Michal
>     >>>
>     >
>
>

Re: Class loading and job versioning

[Stephan Ewen <se...@apache.org>]

Hi Michael!

The only safe way in Java to isolate the user code from the platform would
be to completely run them in different JVMs.

Other than that, Max's method to ensure the correct instantiation should
work in most cases.

We also continuously try to have fewer dependencies in Flink, to that there
is less to clash.

Stephan


On Wed, Jul 20, 2016 at 3:53 PM, Maximilian Michels <mx...@apache.org> wrote:

> Sure. No Problem.
>
> The issue is a bit more involved. You're right, the user classes have
> precedence over the Flink classpath. So your classes were probably
> loaded fine. However, the user code also calls Flink code which can
> use a library version different from the job jar library. And boom, it
> crashes because the job jar library has been loaded previously :)
>
> The only way we could circumvent this problem would be to explicitly
> set a different classloader and load Flink internal classes via
> reflection. I think the performance penalty for this would be way too
> high.
>
> Best,
> Max
>
> On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
> <mi...@zalando.de> wrote:
> > Thanks for the prompt replay.
> >
> > You are right. The conflict was between "com.fasterxml.jackson.core"
> libs.
> >
> > I am just wondering. If the the jobs were separted from the platform,
> >
> > the jobs libs should have precedence and no versioning problem should
> have
> > happened.
> >
> > Regards,
> >
> > Michal
> >
> >
> >
> > On 20.07.2016 14:00, Maximilian Michels wrote:
> >>
> >> Hi Michal,
> >>
> >> I couldn't find Joda in flink-dist. Possibly there is some other clash?
> >>
> >> There are two potential issues here:
> >>
> >> 1) Flink shades some libraries (Guava) but not all. If you use a
> >> version of a library in your Flink job which doesn't match the one in
> >> flink-dist, you're bound for trouble.
> >>
> >> 2) Flink separates jobs from each other to avoid potential class
> >> version mismatches. Each job has its own classloader. In this sense,
> >> "job versioning" is supported.
> >>
> >> Cheers,
> >> Max
> >>
> >> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
> >> <mi...@zalando.de> wrote:
> >>>
> >>> Hi all,
> >>> We had a class versioning problem within Flink Job.
> >>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
> >>> The problem was solved by relocating job classes with shade plug-in.
> >>>
> >>> Does flink separate jobs from each other to avoid class conflicts
> between
> >>> them and the platform ?
> >>> Is job versioning supported or is shading always required ?
> >>>
> >>> Regards,
> >>> Michal
> >>>
> >
>

Re: Class loading and job versioning

[Maximilian Michels <mx...@apache.org>]

Sure. No Problem.

The issue is a bit more involved. You're right, the user classes have
precedence over the Flink classpath. So your classes were probably
loaded fine. However, the user code also calls Flink code which can
use a library version different from the job jar library. And boom, it
crashes because the job jar library has been loaded previously :)

The only way we could circumvent this problem would be to explicitly
set a different classloader and load Flink internal classes via
reflection. I think the performance penalty for this would be way too
high.

Best,
Max

On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
<mi...@zalando.de> wrote:
> Thanks for the prompt replay.
>
> You are right. The conflict was between "com.fasterxml.jackson.core" libs.
>
> I am just wondering. If the the jobs were separted from the platform,
>
> the jobs libs should have precedence and no versioning problem should have
> happened.
>
> Regards,
>
> Michal
>
>
>
> On 20.07.2016 14:00, Maximilian Michels wrote:
>>
>> Hi Michal,
>>
>> I couldn't find Joda in flink-dist. Possibly there is some other clash?
>>
>> There are two potential issues here:
>>
>> 1) Flink shades some libraries (Guava) but not all. If you use a
>> version of a library in your Flink job which doesn't match the one in
>> flink-dist, you're bound for trouble.
>>
>> 2) Flink separates jobs from each other to avoid potential class
>> version mismatches. Each job has its own classloader. In this sense,
>> "job versioning" is supported.
>>
>> Cheers,
>> Max
>>
>> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
>> <mi...@zalando.de> wrote:
>>>
>>> Hi all,
>>> We had a class versioning problem within Flink Job.
>>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>>> The problem was solved by relocating job classes with shade plug-in.
>>>
>>> Does flink separate jobs from each other to avoid class conflicts between
>>> them and the platform ?
>>> Is job versioning supported or is shading always required ?
>>>
>>> Regards,
>>> Michal
>>>
>

Re: Class loading and job versioning

[Michal Budzyn <mi...@zalando.de>]

Thanks for the prompt replay.

You are right. The conflict was between "com.fasterxml.jackson.core" libs.

I am just wondering. If the the jobs were separted from the platform,

the jobs libs should have precedence and no versioning problem should 
have happened.

Regards,

Michal


On 20.07.2016 14:00, Maximilian Michels wrote:
> Hi Michal,
>
> I couldn't find Joda in flink-dist. Possibly there is some other clash?
>
> There are two potential issues here:
>
> 1) Flink shades some libraries (Guava) but not all. If you use a
> version of a library in your Flink job which doesn't match the one in
> flink-dist, you're bound for trouble.
>
> 2) Flink separates jobs from each other to avoid potential class
> version mismatches. Each job has its own classloader. In this sense,
> "job versioning" is supported.
>
> Cheers,
> Max
>
> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
> <mi...@zalando.de> wrote:
>> Hi all,
>> We had a class versioning problem within Flink Job.
>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>> The problem was solved by relocating job classes with shade plug-in.
>>
>> Does flink separate jobs from each other to avoid class conflicts between
>> them and the platform ?
>> Is job versioning supported or is shading always required ?
>>
>> Regards,
>> Michal
>>

Re: Class loading and job versioning

[Maximilian Michels <mx...@apache.org>]

Hi Michal,

I couldn't find Joda in flink-dist. Possibly there is some other clash?

There are two potential issues here:

1) Flink shades some libraries (Guava) but not all. If you use a
version of a library in your Flink job which doesn't match the one in
flink-dist, you're bound for trouble.

2) Flink separates jobs from each other to avoid potential class
version mismatches. Each job has its own classloader. In this sense,
"job versioning" is supported.

Cheers,
Max

On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
<mi...@zalando.de> wrote:
>
> Hi all,
> We had a class versioning problem within Flink Job.
> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
> The problem was solved by relocating job classes with shade plug-in.
>
> Does flink separate jobs from each other to avoid class conflicts between
> them and the platform ?
> Is job versioning supported or is shading always required ?
>
> Regards,
> Michal
>