You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Debasish Ghosh <gh...@gmail.com> on 2019/07/24 06:31:29 UTC

Flink and Akka version incompatibility ..

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
because of version mismatch between Akka that we use and the one that Flink
uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?

Or is there any other alternative way to handle this issue ? I know Flink
1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring
problem down the line with mismatch between the new releases of Akka and
Flink.

regards.

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re: Re: Re: Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

Thanks for the pointer .. yes, we need this ..

On Thu, Jul 25, 2019 at 9:08 AM Haibo Sun <su...@163.com> wrote:

>
> The following JIRA is about the problem you encounter. I think you should
> be very interested in its comments.There does seem to be a problem with
> shading Akka, and Flink is considering isolating the classloader that
> contain Akka and Scala to allow the applications and Flink to use different
> Akka versions.
>
> https://issues.apache.org/jira/browse/FLINK-10903
>
> Best,
> Haibo
>
> At 2019-07-25 00:07:27, "Debasish Ghosh" <gh...@gmail.com> wrote:
>
> Also wanted to check if anyone has ventured into this exercise of shading
> Akka in Flink ..
> Is this something that qualifies as one of the roadmap items in Flink ?
>
> regards.
>
> On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <gh...@gmail.com>
> wrote:
>
>> Hi Haibo - Thanks for the clarification ..
>>
>> regards.
>>
>> On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <su...@163.com> wrote:
>>
>>> Hi  Debasish Ghosh,
>>>
>>> I agree that Flink should shade its Akka.
>>>
>>> Maybe you misunderstood me. I mean, in the absence of official shading
>>> Akka in Flink, the relatively conservative way is to shade Akka of your
>>> application (I concern Flink won't work well after shading its Akka).
>>>
>>> Best,
>>> Haibo
>>>
>>> At 2019-07-24 16:43:28, "Debasish Ghosh" <gh...@gmail.com>
>>> wrote:
>>>
>>> For our application users are expected to work with Akka APIs - hence if
>>> I shade Akka in my application users will need to work with shaded imports
>>> which feels unnatural. With Flink, Akka is an implementation detail and
>>> Flink users are not expected to use Akka APIs. Hence shading will not have
>>> any user level impact.
>>>
>>> Hence the suggestion to shade Akka in Flink rather than the user
>>> application.
>>>
>>> regards.
>>>
>>> On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>> I think it is better to shade all the dependencies of flink so that all
>>>> the projects that use flink won't hit this kind of issue.
>>>>
>>>>
>>>> Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：
>>>>
>>>>> Hi,   Debasish Ghosh
>>>>>
>>>>> I don't know why not shade Akka, maybe it can be shaded. Chesnay may
>>>>> be able to answer that.
>>>>> I recommend to shade Akka dependency of your application because it
>>>>> don't be known what's wrong with shading Flink's Akka.
>>>>>
>>>>> CC  @Chesnay Schepler
>>>>>
>>>>> Best,
>>>>> Haibo
>>>>>
>>>>> At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> The problem that I am facing is with Akka serialization .. Why not
>>>>> shade the whole of Akka ?
>>>>>
>>>>> java.lang.AbstractMethodError:
>>>>>> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
>>>>>> at
>>>>>> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)
>>>>>
>>>>>
>>>>> Akka 2.6 is just around the corner and I don't think Flink will
>>>>> upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur
>>>>> ..
>>>>>
>>>>> regards.
>>>>>
>>>>> On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I can see that we relocate akka's netty, akka uncommon math but also
>>>>>> be curious why Flink doesn't shaded all of akka dependencies...
>>>>>>
>>>>>> Best,
>>>>>> tison.
>>>>>>
>>>>>>
>>>>>> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>>>>>>
>>>>>>> Hello Haibo -
>>>>>>>
>>>>>>> Yes, my application depends on Akka 2.5.
>>>>>>> Just curious, why do you think it's recommended to shade Akka
>>>>>>> version of my application instead of Flink ?
>>>>>>>
>>>>>>> regards.
>>>>>>>
>>>>>>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi  Debasish Ghosh,
>>>>>>>>
>>>>>>>> Does your application have to depend on Akka 2.5? If not, it's a
>>>>>>>> good idea to always keep the Akka version that the application depend on in
>>>>>>>> line with Flink.
>>>>>>>> If you want to try shading Akka dependency, I think that it is more
>>>>>>>> recommended to shade Akka dependency of your application.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Haibo
>>>>>>>>
>>>>>>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello -
>>>>>>>>
>>>>>>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime
>>>>>>>> errors because of version mismatch between Akka that we use and the one
>>>>>>>> that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency
>>>>>>>> with Flink ?
>>>>>>>>
>>>>>>>> Or is there any other alternative way to handle this issue ? I know
>>>>>>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>>>>>>> recurring problem down the line with mismatch between the new releases of
>>>>>>>> Akka and Flink.
>>>>>>>>
>>>>>>>> regards.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Debasish Ghosh
>>>>>>>> http://manning.com/ghosh2
>>>>>>>> http://manning.com/ghosh
>>>>>>>>
>>>>>>>> Twttr: @debasishg
>>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>>> Code: http://github.com/debasishg
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Debasish Ghosh
>>>>>>> http://manning.com/ghosh2
>>>>>>> http://manning.com/ghosh
>>>>>>>
>>>>>>> Twttr: @debasishg
>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>> Code: http://github.com/debasishg
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Debasish Ghosh
>>>>> http://manning.com/ghosh2
>>>>> http://manning.com/ghosh
>>>>>
>>>>> Twttr: @debasishg
>>>>> Blog: http://debasishg.blogspot.com
>>>>> Code: http://github.com/debasishg
>>>>>
>>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>> --
>>> Sent from my iPhone
>>>
>>>
>>
>> --
>> Debasish Ghosh
>> http://manning.com/ghosh2
>> http://manning.com/ghosh
>>
>> Twttr: @debasishg
>> Blog: http://debasishg.blogspot.com
>> Code: http://github.com/debasishg
>>
>
>
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>
>

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re:Re: Re: Re: Flink and Akka version incompatibility ..

Posted by Haibo Sun <su...@163.com>.

The following JIRA is about the problem you encounter. I think you should be very interested in its comments.There does seem to be a problem with shading Akka, and Flink is considering isolating the classloader that contain Akka and Scala to allow the applications and Flink to use different Akka versions.

https://issues.apache.org/jira/browse/FLINK-10903

Best,
Haibo

At 2019-07-25 00:07:27, "Debasish Ghosh" <gh...@gmail.com> wrote:

Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink ..
Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <gh...@gmail.com> wrote:

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <su...@163.com> wrote:

Hi Debasish Ghosh,

I agree that Flink should shade its Akka.

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <gh...@gmail.com> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact.

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：

Hi, Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC @Chesnay Schepler

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:

I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.

Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5.
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:

Hi Debasish Ghosh,

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink.
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Best Regards

Jeff Zhang
--

Sent from my iPhone

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re: Re: Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

Also wanted to check if anyone has ventured into this exercise of shading
Akka in Flink ..
Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <gh...@gmail.com>
wrote:

> Hi Haibo - Thanks for the clarification ..
>
> regards.
>
> On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <su...@163.com> wrote:
>
>> Hi  Debasish Ghosh,
>>
>> I agree that Flink should shade its Akka.
>>
>> Maybe you misunderstood me. I mean, in the absence of official shading
>> Akka in Flink, the relatively conservative way is to shade Akka of your
>> application (I concern Flink won't work well after shading its Akka).
>>
>> Best,
>> Haibo
>>
>> At 2019-07-24 16:43:28, "Debasish Ghosh" <gh...@gmail.com>
>> wrote:
>>
>> For our application users are expected to work with Akka APIs - hence if
>> I shade Akka in my application users will need to work with shaded imports
>> which feels unnatural. With Flink, Akka is an implementation detail and
>> Flink users are not expected to use Akka APIs. Hence shading will not have
>> any user level impact.
>>
>> Hence the suggestion to shade Akka in Flink rather than the user
>> application.
>>
>> regards.
>>
>> On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> I think it is better to shade all the dependencies of flink so that all
>>> the projects that use flink won't hit this kind of issue.
>>>
>>>
>>> Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：
>>>
>>>> Hi,   Debasish Ghosh
>>>>
>>>> I don't know why not shade Akka, maybe it can be shaded. Chesnay may be
>>>> able to answer that.
>>>> I recommend to shade Akka dependency of your application because it
>>>> don't be known what's wrong with shading Flink's Akka.
>>>>
>>>> CC  @Chesnay Schepler
>>>>
>>>> Best,
>>>> Haibo
>>>>
>>>> At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com>
>>>> wrote:
>>>>
>>>> The problem that I am facing is with Akka serialization .. Why not
>>>> shade the whole of Akka ?
>>>>
>>>> java.lang.AbstractMethodError:
>>>>> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
>>>>> at
>>>>> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)
>>>>
>>>>
>>>> Akka 2.6 is just around the corner and I don't think Flink will upgrade
>>>> to Akka 2.6 that soon .. so somehow this problem is bound to recur ..
>>>>
>>>> regards.
>>>>
>>>> On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:
>>>>
>>>>> I can see that we relocate akka's netty, akka uncommon math but also
>>>>> be curious why Flink doesn't shaded all of akka dependencies...
>>>>>
>>>>> Best,
>>>>> tison.
>>>>>
>>>>>
>>>>> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>>>>>
>>>>>> Hello Haibo -
>>>>>>
>>>>>> Yes, my application depends on Akka 2.5.
>>>>>> Just curious, why do you think it's recommended to shade Akka version
>>>>>> of my application instead of Flink ?
>>>>>>
>>>>>> regards.
>>>>>>
>>>>>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi  Debasish Ghosh,
>>>>>>>
>>>>>>> Does your application have to depend on Akka 2.5? If not, it's a
>>>>>>> good idea to always keep the Akka version that the application depend on in
>>>>>>> line with Flink.
>>>>>>> If you want to try shading Akka dependency, I think that it is more
>>>>>>> recommended to shade Akka dependency of your application.
>>>>>>>
>>>>>>> Best,
>>>>>>> Haibo
>>>>>>>
>>>>>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello -
>>>>>>>
>>>>>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime
>>>>>>> errors because of version mismatch between Akka that we use and the one
>>>>>>> that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency
>>>>>>> with Flink ?
>>>>>>>
>>>>>>> Or is there any other alternative way to handle this issue ? I know
>>>>>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>>>>>> recurring problem down the line with mismatch between the new releases of
>>>>>>> Akka and Flink.
>>>>>>>
>>>>>>> regards.
>>>>>>>
>>>>>>> --
>>>>>>> Debasish Ghosh
>>>>>>> http://manning.com/ghosh2
>>>>>>> http://manning.com/ghosh
>>>>>>>
>>>>>>> Twttr: @debasishg
>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>> Code: http://github.com/debasishg
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Debasish Ghosh
>>>>>> http://manning.com/ghosh2
>>>>>> http://manning.com/ghosh
>>>>>>
>>>>>> Twttr: @debasishg
>>>>>> Blog: http://debasishg.blogspot.com
>>>>>> Code: http://github.com/debasishg
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Debasish Ghosh
>>>> http://manning.com/ghosh2
>>>> http://manning.com/ghosh
>>>>
>>>> Twttr: @debasishg
>>>> Blog: http://debasishg.blogspot.com
>>>> Code: http://github.com/debasishg
>>>>
>>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>> --
>> Sent from my iPhone
>>
>>
>
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>


-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re: Re: Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <su...@163.com> wrote:

> Hi  Debasish Ghosh,
>
> I agree that Flink should shade its Akka.
>
> Maybe you misunderstood me. I mean, in the absence of official shading
> Akka in Flink, the relatively conservative way is to shade Akka of your
> application (I concern Flink won't work well after shading its Akka).
>
> Best,
> Haibo
>
> At 2019-07-24 16:43:28, "Debasish Ghosh" <gh...@gmail.com> wrote:
>
> For our application users are expected to work with Akka APIs - hence if I
> shade Akka in my application users will need to work with shaded imports
> which feels unnatural. With Flink, Akka is an implementation detail and
> Flink users are not expected to use Akka APIs. Hence shading will not have
> any user level impact.
>
> Hence the suggestion to shade Akka in Flink rather than the user
> application.
>
> regards.
>
> On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>> I think it is better to shade all the dependencies of flink so that all
>> the projects that use flink won't hit this kind of issue.
>>
>>
>> Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：
>>
>>> Hi,   Debasish Ghosh
>>>
>>> I don't know why not shade Akka, maybe it can be shaded. Chesnay may be
>>> able to answer that.
>>> I recommend to shade Akka dependency of your application because it
>>> don't be known what's wrong with shading Flink's Akka.
>>>
>>> CC  @Chesnay Schepler
>>>
>>> Best,
>>> Haibo
>>>
>>> At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com>
>>> wrote:
>>>
>>> The problem that I am facing is with Akka serialization .. Why not shade
>>> the whole of Akka ?
>>>
>>> java.lang.AbstractMethodError:
>>>> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
>>>> at
>>>> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)
>>>
>>>
>>> Akka 2.6 is just around the corner and I don't think Flink will upgrade
>>> to Akka 2.6 that soon .. so somehow this problem is bound to recur ..
>>>
>>> regards.
>>>
>>> On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:
>>>
>>>> I can see that we relocate akka's netty, akka uncommon math but also
>>>> be curious why Flink doesn't shaded all of akka dependencies...
>>>>
>>>> Best,
>>>> tison.
>>>>
>>>>
>>>> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>>>>
>>>>> Hello Haibo -
>>>>>
>>>>> Yes, my application depends on Akka 2.5.
>>>>> Just curious, why do you think it's recommended to shade Akka version
>>>>> of my application instead of Flink ?
>>>>>
>>>>> regards.
>>>>>
>>>>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:
>>>>>
>>>>>> Hi  Debasish Ghosh,
>>>>>>
>>>>>> Does your application have to depend on Akka 2.5? If not, it's a
>>>>>> good idea to always keep the Akka version that the application depend on in
>>>>>> line with Flink.
>>>>>> If you want to try shading Akka dependency, I think that it is more
>>>>>> recommended to shade Akka dependency of your application.
>>>>>>
>>>>>> Best,
>>>>>> Haibo
>>>>>>
>>>>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hello -
>>>>>>
>>>>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime
>>>>>> errors because of version mismatch between Akka that we use and the one
>>>>>> that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency
>>>>>> with Flink ?
>>>>>>
>>>>>> Or is there any other alternative way to handle this issue ? I know
>>>>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>>>>> recurring problem down the line with mismatch between the new releases of
>>>>>> Akka and Flink.
>>>>>>
>>>>>> regards.
>>>>>>
>>>>>> --
>>>>>> Debasish Ghosh
>>>>>> http://manning.com/ghosh2
>>>>>> http://manning.com/ghosh
>>>>>>
>>>>>> Twttr: @debasishg
>>>>>> Blog: http://debasishg.blogspot.com
>>>>>> Code: http://github.com/debasishg
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Debasish Ghosh
>>>>> http://manning.com/ghosh2
>>>>> http://manning.com/ghosh
>>>>>
>>>>> Twttr: @debasishg
>>>>> Blog: http://debasishg.blogspot.com
>>>>> Code: http://github.com/debasishg
>>>>>
>>>>
>>>
>>> --
>>> Debasish Ghosh
>>> http://manning.com/ghosh2
>>> http://manning.com/ghosh
>>>
>>> Twttr: @debasishg
>>> Blog: http://debasishg.blogspot.com
>>> Code: http://github.com/debasishg
>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
> --
> Sent from my iPhone
>
>

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re:Re: Re: Flink and Akka version incompatibility ..

Posted by Haibo Sun <su...@163.com>.

Hi Debasish Ghosh,

I agree that Flink should shade its Akka.

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <gh...@gmail.com> wrote:

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：

Hi, Debasish Ghosh

CC @Chesnay Schepler

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:

I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.

Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5.
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:

Hi Debasish Ghosh,

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com> wrote:

Hello -

regards.

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Best Regards

Jeff Zhang
--

Sent from my iPhone

Re: Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

For our application users are expected to work with Akka APIs - hence if I
shade Akka in my application users will need to work with shaded imports
which feels unnatural. With Flink, Akka is an implementation detail and
Flink users are not expected to use Akka APIs. Hence shading will not have
any user level impact.

Hence the suggestion to shade Akka in Flink rather than the user
application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <zj...@gmail.com> wrote:

> I think it is better to shade all the dependencies of flink so that all
> the projects that use flink won't hit this kind of issue.
>
>
> Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：
>
>> Hi,   Debasish Ghosh
>>
>> I don't know why not shade Akka, maybe it can be shaded. Chesnay may be
>> able to answer that.
>> I recommend to shade Akka dependency of your application because it don't
>> be known what's wrong with shading Flink's Akka.
>>
>> CC  @Chesnay Schepler
>>
>> Best,
>> Haibo
>>
>> At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com>
>> wrote:
>>
>> The problem that I am facing is with Akka serialization .. Why not shade
>> the whole of Akka ?
>>
>> java.lang.AbstractMethodError:
>>> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
>>> at
>>> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)
>>
>>
>> Akka 2.6 is just around the corner and I don't think Flink will upgrade
>> to Akka 2.6 that soon .. so somehow this problem is bound to recur ..
>>
>> regards.
>>
>> On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:
>>
>>> I can see that we relocate akka's netty, akka uncommon math but also
>>> be curious why Flink doesn't shaded all of akka dependencies...
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>>>
>>>> Hello Haibo -
>>>>
>>>> Yes, my application depends on Akka 2.5.
>>>> Just curious, why do you think it's recommended to shade Akka version
>>>> of my application instead of Flink ?
>>>>
>>>> regards.
>>>>
>>>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:
>>>>
>>>>> Hi  Debasish Ghosh,
>>>>>
>>>>> Does your application have to depend on Akka 2.5? If not, it's a good
>>>>> idea to always keep the Akka version that the application depend on in line
>>>>> with Flink.
>>>>> If you want to try shading Akka dependency, I think that it is more
>>>>> recommended to shade Akka dependency of your application.
>>>>>
>>>>> Best,
>>>>> Haibo
>>>>>
>>>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hello -
>>>>>
>>>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
>>>>> because of version mismatch between Akka that we use and the one that Flink
>>>>> uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?
>>>>>
>>>>> Or is there any other alternative way to handle this issue ? I know
>>>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>>>> recurring problem down the line with mismatch between the new releases of
>>>>> Akka and Flink.
>>>>>
>>>>> regards.
>>>>>
>>>>> --
>>>>> Debasish Ghosh
>>>>> http://manning.com/ghosh2
>>>>> http://manning.com/ghosh
>>>>>
>>>>> Twttr: @debasishg
>>>>> Blog: http://debasishg.blogspot.com
>>>>> Code: http://github.com/debasishg
>>>>>
>>>>>
>>>>
>>>> --
>>>> Debasish Ghosh
>>>> http://manning.com/ghosh2
>>>> http://manning.com/ghosh
>>>>
>>>> Twttr: @debasishg
>>>> Blog: http://debasishg.blogspot.com
>>>> Code: http://github.com/debasishg
>>>>
>>>
>>
>> --
>> Debasish Ghosh
>> http://manning.com/ghosh2
>> http://manning.com/ghosh
>>
>> Twttr: @debasishg
>> Blog: http://debasishg.blogspot.com
>> Code: http://github.com/debasishg
>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
-- 
Sent from my iPhone

Re: Re: Flink and Akka version incompatibility ..

Posted by Jeff Zhang <zj...@gmail.com>.

I think it is better to shade all the dependencies of flink so that all the
projects that use flink won't hit this kind of issue.


Haibo Sun <su...@163.com> 于2019年7月24日周三 下午4:07写道：

> Hi,   Debasish Ghosh
>
> I don't know why not shade Akka, maybe it can be shaded. Chesnay may be
> able to answer that.
> I recommend to shade Akka dependency of your application because it don't
> be known what's wrong with shading Flink's Akka.
>
> CC  @Chesnay Schepler
>
> Best,
> Haibo
>
> At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com> wrote:
>
> The problem that I am facing is with Akka serialization .. Why not shade
> the whole of Akka ?
>
> java.lang.AbstractMethodError:
>> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
>> at
>> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)
>
>
> Akka 2.6 is just around the corner and I don't think Flink will upgrade to
> Akka 2.6 that soon .. so somehow this problem is bound to recur ..
>
> regards.
>
> On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:
>
>> I can see that we relocate akka's netty, akka uncommon math but also
>> be curious why Flink doesn't shaded all of akka dependencies...
>>
>> Best,
>> tison.
>>
>>
>> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>>
>>> Hello Haibo -
>>>
>>> Yes, my application depends on Akka 2.5.
>>> Just curious, why do you think it's recommended to shade Akka version of
>>> my application instead of Flink ?
>>>
>>> regards.
>>>
>>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:
>>>
>>>> Hi  Debasish Ghosh,
>>>>
>>>> Does your application have to depend on Akka 2.5? If not, it's a good
>>>> idea to always keep the Akka version that the application depend on in line
>>>> with Flink.
>>>> If you want to try shading Akka dependency, I think that it is more
>>>> recommended to shade Akka dependency of your application.
>>>>
>>>> Best,
>>>> Haibo
>>>>
>>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>>> wrote:
>>>>
>>>> Hello -
>>>>
>>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
>>>> because of version mismatch between Akka that we use and the one that Flink
>>>> uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?
>>>>
>>>> Or is there any other alternative way to handle this issue ? I know
>>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>>> recurring problem down the line with mismatch between the new releases of
>>>> Akka and Flink.
>>>>
>>>> regards.
>>>>
>>>> --
>>>> Debasish Ghosh
>>>> http://manning.com/ghosh2
>>>> http://manning.com/ghosh
>>>>
>>>> Twttr: @debasishg
>>>> Blog: http://debasishg.blogspot.com
>>>> Code: http://github.com/debasishg
>>>>
>>>>
>>>
>>> --
>>> Debasish Ghosh
>>> http://manning.com/ghosh2
>>> http://manning.com/ghosh
>>>
>>> Twttr: @debasishg
>>> Blog: http://debasishg.blogspot.com
>>> Code: http://github.com/debasishg
>>>
>>
>
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>
>

-- 
Best Regards

Jeff Zhang

Re:Re: Flink and Akka version incompatibility ..

Posted by Haibo Sun <su...@163.com>.

Hi, Debasish Ghosh

CC @Chesnay Schepler

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <gh...@gmail.com> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:

I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.

Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5.
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:

Hi Debasish Ghosh,

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com> wrote:

Hello -

regards.

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

The problem that I am facing is with Akka serialization .. Why not shade
the whole of Akka ?

java.lang.AbstractMethodError:
> akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
> at
> akka.serialization.Serialization.serializationInformation(Serialization.scala:166)


Akka 2.6 is just around the corner and I don't think Flink will upgrade to
Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <wa...@gmail.com> wrote:

> I can see that we relocate akka's netty, akka uncommon math but also
> be curious why Flink doesn't shaded all of akka dependencies...
>
> Best,
> tison.
>
>
> Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：
>
>> Hello Haibo -
>>
>> Yes, my application depends on Akka 2.5.
>> Just curious, why do you think it's recommended to shade Akka version of
>> my application instead of Flink ?
>>
>> regards.
>>
>> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:
>>
>>> Hi  Debasish Ghosh,
>>>
>>> Does your application have to depend on Akka 2.5? If not, it's a good
>>> idea to always keep the Akka version that the application depend on in line
>>> with Flink.
>>> If you want to try shading Akka dependency, I think that it is more
>>> recommended to shade Akka dependency of your application.
>>>
>>> Best,
>>> Haibo
>>>
>>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>>> wrote:
>>>
>>> Hello -
>>>
>>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
>>> because of version mismatch between Akka that we use and the one that Flink
>>> uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?
>>>
>>> Or is there any other alternative way to handle this issue ? I know
>>> Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a
>>> recurring problem down the line with mismatch between the new releases of
>>> Akka and Flink.
>>>
>>> regards.
>>>
>>> --
>>> Debasish Ghosh
>>> http://manning.com/ghosh2
>>> http://manning.com/ghosh
>>>
>>> Twttr: @debasishg
>>> Blog: http://debasishg.blogspot.com
>>> Code: http://github.com/debasishg
>>>
>>>
>>
>> --
>> Debasish Ghosh
>> http://manning.com/ghosh2
>> http://manning.com/ghosh
>>
>> Twttr: @debasishg
>> Blog: http://debasishg.blogspot.com
>> Code: http://github.com/debasishg
>>
>

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re: Flink and Akka version incompatibility ..

Posted by Zili Chen <wa...@gmail.com>.

I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <gh...@gmail.com> 于2019年7月24日周三 下午3:15写道：

> Hello Haibo -
>
> Yes, my application depends on Akka 2.5.
> Just curious, why do you think it's recommended to shade Akka version of
> my application instead of Flink ?
>
> regards.
>
> On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:
>
>> Hi  Debasish Ghosh,
>>
>> Does your application have to depend on Akka 2.5? If not, it's a good
>> idea to always keep the Akka version that the application depend on in line
>> with Flink.
>> If you want to try shading Akka dependency, I think that it is more
>> recommended to shade Akka dependency of your application.
>>
>> Best,
>> Haibo
>>
>> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com>
>> wrote:
>>
>> Hello -
>>
>> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
>> because of version mismatch between Akka that we use and the one that Flink
>> uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?
>>
>> Or is there any other alternative way to handle this issue ? I know Flink
>> 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring
>> problem down the line with mismatch between the new releases of Akka and
>> Flink.
>>
>> regards.
>>
>> --
>> Debasish Ghosh
>> http://manning.com/ghosh2
>> http://manning.com/ghosh
>>
>> Twttr: @debasishg
>> Blog: http://debasishg.blogspot.com
>> Code: http://github.com/debasishg
>>
>>
>
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>

Re: Flink and Akka version incompatibility ..

Posted by Debasish Ghosh <gh...@gmail.com>.

Hello Haibo -

Yes, my application depends on Akka 2.5.
Just curious, why do you think it's recommended to shade Akka version of my
application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <su...@163.com> wrote:

> Hi  Debasish Ghosh,
>
> Does your application have to depend on Akka 2.5? If not, it's a good
> idea to always keep the Akka version that the application depend on in line
> with Flink.
> If you want to try shading Akka dependency, I think that it is more
> recommended to shade Akka dependency of your application.
>
> Best,
> Haibo
>
> At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com> wrote:
>
> Hello -
>
> An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors
> because of version mismatch between Akka that we use and the one that Flink
> uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ?
>
> Or is there any other alternative way to handle this issue ? I know Flink
> 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring
> problem down the line with mismatch between the new releases of Akka and
> Flink.
>
> regards.
>
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>
>

-- 
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

Re:Flink and Akka version incompatibility ..

Posted by Haibo Sun <su...@163.com>.

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.


Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <gh...@gmail.com> wrote:

Hello -


An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 


Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.


regards.



--

Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh


Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg