You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Jangho Seo <ja...@jangho.kr> on 2020/02/01 08:13:18 UTC
Re: Help migrating Netty 4.0 -> 4.1
I've come up with a pull request [1].
Also, I'm attaching the patch for future references.
Please test it and let me know if anything is wrong.
Thanks,
Jangho
[1] https://github.com/motus/reef/pull/1
On 1/29/20 12:15 PM, Sergiy Matusevych wrote:
> Thank you! I'm looking forward for your pull request :)
>
> Cheers,
> Sergiy.
>
> On Tue, Jan 28, 2020 at 7:06 PM Jangho Seo <jangho@jangho.kr
> <ma...@jangho.kr>> wrote:
>
> Hi Sergiy,
>
> I was able to reproduce the issue. I'll continue to investigate.
>
> Thanks,
> Jangho
>
> On 1/28/20 8:20 AM, Sergiy Matusevych wrote:
>> Thank you so much Gon and the team!
>>
>> I got really stuck with this one last weekend. It looks like
>> upgrading other dependencies is trivial, except of Netty and
>> Checkstyle/Findbugs. It'd be great if you guys can help us with
>> Netty while I take care of the rest.
>>
>> Thank you,
>> Sergiy.
>>
>>
>> On Mon, Jan 27, 2020 at 3:04 PM Byung-Gon Chun <bgchun@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi Sergiy,
>>
>> Thanks for the effort to upgrade Netty from 4.0 to 4.1.
>> I'm adding Taegeon and Jangho, who know the codebase pretty
>> well.
>>
>> Thanks.
>> -Gon
>>
>> On Tue, Jan 28, 2020 at 5:22 AM Sergiy Matusevych
>> <sergiy.matusevych@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi REEF team,
>>
>> I need your help investigating Wake unit test failures
>> when using Netty 4.1.
>>
>> Here's what's going on. I am updating REEF dependencies
>> to make it more in sync with Hadoop and Spark, among
>> other things. One of the packages that we use is Netty
>> 4.0, and I want to switch it to version 4.1.42.Final.
>> (Besides, we already use Netty 4.1 in our gRPC bridge).
>> Unfortunately, drop-in replacement does not seem to work
>> - there are two unit tests in Wake that consistently fail
>> under Netty 4.1.
>>
>> I've described the situation in this JIRA issue:
>> https://issues.apache.org/jira/browse/REEF-2059
>>
>> Looks like it has something to do with reconnect?
>>
>> Very likely, there are subtle changes in Netty behavior
>> that we have to take into account in our code. Can
>> someone more familiar with Wake take a look at the issue?
>>
>> To reproduce, do the following:
>>
>> 1) fork branch REEF-2057 from my github repo at
>> https://github.com/motus/reef/tree/REEF-2057
>>
>> 2) cd ./lang/java/reef-wake
>>
>> 3) run
>> mvn test
>> -Dtest='org.apache.reef.wake.test.remote.RemoteManagerTest'
>>
>> Gon, maybe you or someone from your team can take a look
>> at that?
>>
>> Meanwhile, I'll continue to upgrade other dependencies,
>> and I hope we can do a minor release when we're done.
>>
>> Cheers,
>> Sergiy.
>>
>>
>>
>> --
>> Byung-Gon Chun
>>
Re: Help migrating Netty 4.0 -> 4.1
Posted by Sergiy Matusevych <se...@gmail.com>.
Hi Jangho,
That's right, I totally forgot that we're still Java 1.7 project :) Let's
discuss that in a separate thread - I think it is time to embrace the shiny
new Java 1.8 :)
Thank you,
Sergiy.
On Sat, Feb 1, 2020 at 9:38 PM Jangho Seo <ja...@jangho.kr> wrote:
> Great work, Sergiy!
>
> It's nice to see that all properties for versions are neatly defined in
> the top-level pom.xml [1]. And the dependency tree [2] confirms that Netty
> 4.1.42.Final is the only version in the Netty 4.x series in our dependency
> set. Nice!
>
> One concern is that upgrading Hadoop to 3.x means that we're dropping
> support for Java 7 [3]. I can see that Travis is complaining about that.
>
> Thank you,
> Jangho
>
> [1] With one exception of protobuf.version=3.5.1 in some child pom.xmls,
> with a good and known reason.
> [2]
> https://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html
> [3]
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
>
Re: Help migrating Netty 4.0 -> 4.1
Posted by Jangho Seo <ja...@jangho.kr>.
Great work, Sergiy!
It's nice to see that all properties for versions are neatly defined in
the top-level pom.xml [1]. And the dependency tree [2] confirms that
Netty 4.1.42.Final is the only version in the Netty 4.x series in our
dependency set. Nice!
One concern is that upgrading Hadoop to 3.x means that we're dropping
support for Java 7 [3]. I can see that Travis is complaining about that.
Thank you,
Jangho
[1] With one exception of protobuf.version=3.5.1 in some child pom.xmls,
with a good and known reason.
[2] https://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html
[3] https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
On 2/2/20 1:48 PM, Sergiy Matusevych wrote:
> Totally agree. Now we have to fix the rest of the stuff :)
>
> So, Jangho, while you are at it, maybe you can take a look at the PR
> with your Netty fix + other dependencies updated?
> Here's the link: https://github.com/apache/reef/pull/1496
>
> Thank you!
> Sergiy.
>
>
>
> On Sat, Feb 1, 2020 at 8:42 PM Jangho Seo <jangho@jangho.kr
> <ma...@jangho.kr>> wrote:
>
> Glad to hear it worked. The benefit of having one version of Netty
> as a dependency looks great.
>
> Thank you,
> Jangho
>
> On 2/2/20 11:14 AM, Sergiy Matusevych wrote:
>> Thank you so much Jangho! That was really helpful, and now I am
>> ready to roll out my PR with new dependencies! One good thing
>> about the fis is that now we don't need to have two different
>> versions of Netty for gRPC bridge and the rest of REEF, and I was
>> able to remove Netty shadowing in the bridge. I'll push a new PR
>> soon.
>>
>> Thank you,
>> Sergiy.
>>
>> On Sat, Feb 1, 2020 at 12:13 AM Jangho Seo <jangho@jangho.kr
>> <ma...@jangho.kr>> wrote:
>>
>> I've come up with a pull request [1].
>> Also, I'm attaching the patch for future references.
>>
>> Please test it and let me know if anything is wrong.
>>
>> Thanks,
>> Jangho
>>
>> [1] https://github.com/motus/reef/pull/1
>>
>> On 1/29/20 12:15 PM, Sergiy Matusevych wrote:
>>> Thank you! I'm looking forward for your pull request :)
>>>
>>> Cheers,
>>> Sergiy.
>>>
>>> On Tue, Jan 28, 2020 at 7:06 PM Jangho Seo <jangho@jangho.kr
>>> <ma...@jangho.kr>> wrote:
>>>
>>> Hi Sergiy,
>>>
>>> I was able to reproduce the issue. I'll continue to
>>> investigate.
>>>
>>> Thanks,
>>> Jangho
>>>
>>> On 1/28/20 8:20 AM, Sergiy Matusevych wrote:
>>>> Thank you so much Gon and the team!
>>>>
>>>> I got really stuck with this one last weekend. It looks
>>>> like upgrading other dependencies is trivial, except of
>>>> Netty and Checkstyle/Findbugs. It'd be great if you
>>>> guys can help us with Netty while I take care of the rest.
>>>>
>>>> Thank you,
>>>> Sergiy.
>>>>
>>>>
>>>> On Mon, Jan 27, 2020 at 3:04 PM Byung-Gon Chun
>>>> <bgchun@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hi Sergiy,
>>>>
>>>> Thanks for the effort to upgrade Netty from 4.0 to 4.1.
>>>> I'm adding Taegeon and Jangho, who know the
>>>> codebase pretty well.
>>>>
>>>> Thanks.
>>>> -Gon
>>>>
>>>> On Tue, Jan 28, 2020 at 5:22 AM Sergiy Matusevych
>>>> <sergiy.matusevych@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>
>>>> Hi REEF team,
>>>>
>>>> I need your help investigating Wake unit test
>>>> failures when using Netty 4.1.
>>>>
>>>> Here's what's going on. I am updating REEF
>>>> dependencies to make it more in sync with
>>>> Hadoop and Spark, among other things. One of
>>>> the packages that we use is Netty 4.0, and I
>>>> want to switch it to version 4.1.42.Final.
>>>> (Besides, we already use Netty 4.1 in our gRPC
>>>> bridge). Unfortunately, drop-in replacement
>>>> does not seem to work - there are two unit
>>>> tests in Wake that consistently fail under
>>>> Netty 4.1.
>>>>
>>>> I've described the situation in this JIRA issue:
>>>> https://issues.apache.org/jira/browse/REEF-2059
>>>>
>>>> Looks like it has something to do with reconnect?
>>>>
>>>> Very likely, there are subtle changes in Netty
>>>> behavior that we have to take into account in
>>>> our code. Can someone more familiar with Wake
>>>> take a look at the issue?
>>>>
>>>> To reproduce, do the following:
>>>>
>>>> 1) fork branch REEF-2057 from my github repo at
>>>> https://github.com/motus/reef/tree/REEF-2057
>>>>
>>>> 2) cd ./lang/java/reef-wake
>>>>
>>>> 3) run
>>>> mvn test
>>>> -Dtest='org.apache.reef.wake.test.remote.RemoteManagerTest'
>>>>
>>>> Gon, maybe you or someone from your team can
>>>> take a look at that?
>>>>
>>>> Meanwhile, I'll continue to upgrade other
>>>> dependencies, and I hope we can do a minor
>>>> release when we're done.
>>>>
>>>> Cheers,
>>>> Sergiy.
>>>>
>>>>
>>>>
>>>> --
>>>> Byung-Gon Chun
>>>>
Re: Help migrating Netty 4.0 -> 4.1
Posted by Sergiy Matusevych <se...@gmail.com>.
Totally agree. Now we have to fix the rest of the stuff :)
So, Jangho, while you are at it, maybe you can take a look at the PR with
your Netty fix + other dependencies updated?
Here's the link: https://github.com/apache/reef/pull/1496
Thank you!
Sergiy.
On Sat, Feb 1, 2020 at 8:42 PM Jangho Seo <ja...@jangho.kr> wrote:
> Glad to hear it worked. The benefit of having one version of Netty as a
> dependency looks great.
>
> Thank you,
> Jangho
> On 2/2/20 11:14 AM, Sergiy Matusevych wrote:
>
> Thank you so much Jangho! That was really helpful, and now I am ready to
> roll out my PR with new dependencies! One good thing about the fis is that
> now we don't need to have two different versions of Netty for gRPC bridge
> and the rest of REEF, and I was able to remove Netty shadowing in the
> bridge. I'll push a new PR soon.
>
> Thank you,
> Sergiy.
>
> On Sat, Feb 1, 2020 at 12:13 AM Jangho Seo <ja...@jangho.kr> wrote:
>
>> I've come up with a pull request [1].
>> Also, I'm attaching the patch for future references.
>>
>> Please test it and let me know if anything is wrong.
>>
>> Thanks,
>> Jangho
>>
>> [1] https://github.com/motus/reef/pull/1
>> On 1/29/20 12:15 PM, Sergiy Matusevych wrote:
>>
>> Thank you! I'm looking forward for your pull request :)
>>
>> Cheers,
>> Sergiy.
>>
>> On Tue, Jan 28, 2020 at 7:06 PM Jangho Seo <ja...@jangho.kr> wrote:
>>
>>> Hi Sergiy,
>>>
>>> I was able to reproduce the issue. I'll continue to investigate.
>>>
>>> Thanks,
>>> Jangho
>>> On 1/28/20 8:20 AM, Sergiy Matusevych wrote:
>>>
>>> Thank you so much Gon and the team!
>>>
>>> I got really stuck with this one last weekend. It looks like upgrading
>>> other dependencies is trivial, except of Netty and Checkstyle/Findbugs.
>>> It'd be great if you guys can help us with Netty while I take care of the
>>> rest.
>>>
>>> Thank you,
>>> Sergiy.
>>>
>>>
>>> On Mon, Jan 27, 2020 at 3:04 PM Byung-Gon Chun <bg...@gmail.com> wrote:
>>>
>>>> Hi Sergiy,
>>>>
>>>> Thanks for the effort to upgrade Netty from 4.0 to 4.1.
>>>> I'm adding Taegeon and Jangho, who know the codebase pretty well.
>>>>
>>>> Thanks.
>>>> -Gon
>>>>
>>>> On Tue, Jan 28, 2020 at 5:22 AM Sergiy Matusevych <
>>>> sergiy.matusevych@gmail.com> wrote:
>>>>
>>>>> Hi REEF team,
>>>>>
>>>>> I need your help investigating Wake unit test failures when using
>>>>> Netty 4.1.
>>>>>
>>>>> Here's what's going on. I am updating REEF dependencies to make it
>>>>> more in sync with Hadoop and Spark, among other things. One of the packages
>>>>> that we use is Netty 4.0, and I want to switch it to version 4.1.42.Final.
>>>>> (Besides, we already use Netty 4.1 in our gRPC bridge). Unfortunately,
>>>>> drop-in replacement does not seem to work - there are two unit tests in
>>>>> Wake that consistently fail under Netty 4.1.
>>>>>
>>>>> I've described the situation in this JIRA issue:
>>>>> https://issues.apache.org/jira/browse/REEF-2059
>>>>>
>>>>> Looks like it has something to do with reconnect?
>>>>>
>>>>> Very likely, there are subtle changes in Netty behavior that we have
>>>>> to take into account in our code. Can someone more familiar with Wake take
>>>>> a look at the issue?
>>>>>
>>>>> To reproduce, do the following:
>>>>>
>>>>> 1) fork branch REEF-2057 from my github repo at
>>>>> https://github.com/motus/reef/tree/REEF-2057
>>>>>
>>>>> 2) cd ./lang/java/reef-wake
>>>>>
>>>>> 3) run
>>>>> mvn test -Dtest='org.apache.reef.wake.test.remote.RemoteManagerTest'
>>>>>
>>>>> Gon, maybe you or someone from your team can take a look at that?
>>>>>
>>>>> Meanwhile, I'll continue to upgrade other dependencies, and I hope we
>>>>> can do a minor release when we're done.
>>>>>
>>>>> Cheers,
>>>>> Sergiy.
>>>>>
>>>>>
>>>>
>>>> --
>>>> Byung-Gon Chun
>>>>
>>>
Re: Help migrating Netty 4.0 -> 4.1
Posted by Jangho Seo <ja...@jangho.kr>.
Glad to hear it worked. The benefit of having one version of Netty as a
dependency looks great.
Thank you,
Jangho
On 2/2/20 11:14 AM, Sergiy Matusevych wrote:
> Thank you so much Jangho! That was really helpful, and now I am ready
> to roll out my PR with new dependencies! One good thing about the fis
> is that now we don't need to have two different versions of Netty for
> gRPC bridge and the rest of REEF, and I was able to remove Netty
> shadowing in the bridge. I'll push a new PR soon.
>
> Thank you,
> Sergiy.
>
> On Sat, Feb 1, 2020 at 12:13 AM Jangho Seo <jangho@jangho.kr
> <ma...@jangho.kr>> wrote:
>
> I've come up with a pull request [1].
> Also, I'm attaching the patch for future references.
>
> Please test it and let me know if anything is wrong.
>
> Thanks,
> Jangho
>
> [1] https://github.com/motus/reef/pull/1
>
> On 1/29/20 12:15 PM, Sergiy Matusevych wrote:
>> Thank you! I'm looking forward for your pull request :)
>>
>> Cheers,
>> Sergiy.
>>
>> On Tue, Jan 28, 2020 at 7:06 PM Jangho Seo <jangho@jangho.kr
>> <ma...@jangho.kr>> wrote:
>>
>> Hi Sergiy,
>>
>> I was able to reproduce the issue. I'll continue to investigate.
>>
>> Thanks,
>> Jangho
>>
>> On 1/28/20 8:20 AM, Sergiy Matusevych wrote:
>>> Thank you so much Gon and the team!
>>>
>>> I got really stuck with this one last weekend. It looks like
>>> upgrading other dependencies is trivial, except of Netty and
>>> Checkstyle/Findbugs. It'd be great if you guys can help us
>>> with Netty while I take care of the rest.
>>>
>>> Thank you,
>>> Sergiy.
>>>
>>>
>>> On Mon, Jan 27, 2020 at 3:04 PM Byung-Gon Chun
>>> <bgchun@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Sergiy,
>>>
>>> Thanks for the effort to upgrade Netty from 4.0 to 4.1.
>>> I'm adding Taegeon and Jangho, who know the codebase
>>> pretty well.
>>>
>>> Thanks.
>>> -Gon
>>>
>>> On Tue, Jan 28, 2020 at 5:22 AM Sergiy Matusevych
>>> <sergiy.matusevych@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hi REEF team,
>>>
>>> I need your help investigating Wake unit test
>>> failures when using Netty 4.1.
>>>
>>> Here's what's going on. I am updating REEF
>>> dependencies to make it more in sync with Hadoop and
>>> Spark, among other things. One of the packages that
>>> we use is Netty 4.0, and I want to switch it to
>>> version 4.1.42.Final. (Besides, we already use Netty
>>> 4.1 in our gRPC bridge). Unfortunately, drop-in
>>> replacement does not seem to work - there are two
>>> unit tests in Wake that consistently fail under
>>> Netty 4.1.
>>>
>>> I've described the situation in this JIRA issue:
>>> https://issues.apache.org/jira/browse/REEF-2059
>>>
>>> Looks like it has something to do with reconnect?
>>>
>>> Very likely, there are subtle changes in Netty
>>> behavior that we have to take into account in our
>>> code. Can someone more familiar with Wake take a
>>> look at the issue?
>>>
>>> To reproduce, do the following:
>>>
>>> 1) fork branch REEF-2057 from my github repo at
>>> https://github.com/motus/reef/tree/REEF-2057
>>>
>>> 2) cd ./lang/java/reef-wake
>>>
>>> 3) run
>>> mvn test
>>> -Dtest='org.apache.reef.wake.test.remote.RemoteManagerTest'
>>>
>>> Gon, maybe you or someone from your team can take a
>>> look at that?
>>>
>>> Meanwhile, I'll continue to upgrade other
>>> dependencies, and I hope we can do a minor release
>>> when we're done.
>>>
>>> Cheers,
>>> Sergiy.
>>>
>>>
>>>
>>> --
>>> Byung-Gon Chun
>>>
Re: Help migrating Netty 4.0 -> 4.1
Posted by Sergiy Matusevych <se...@gmail.com>.
Thank you so much Jangho! That was really helpful, and now I am ready to
roll out my PR with new dependencies! One good thing about the fis is that
now we don't need to have two different versions of Netty for gRPC bridge
and the rest of REEF, and I was able to remove Netty shadowing in the
bridge. I'll push a new PR soon.
Thank you,
Sergiy.
On Sat, Feb 1, 2020 at 12:13 AM Jangho Seo <ja...@jangho.kr> wrote:
> I've come up with a pull request [1].
> Also, I'm attaching the patch for future references.
>
> Please test it and let me know if anything is wrong.
>
> Thanks,
> Jangho
>
> [1] https://github.com/motus/reef/pull/1
> On 1/29/20 12:15 PM, Sergiy Matusevych wrote:
>
> Thank you! I'm looking forward for your pull request :)
>
> Cheers,
> Sergiy.
>
> On Tue, Jan 28, 2020 at 7:06 PM Jangho Seo <ja...@jangho.kr> wrote:
>
>> Hi Sergiy,
>>
>> I was able to reproduce the issue. I'll continue to investigate.
>>
>> Thanks,
>> Jangho
>> On 1/28/20 8:20 AM, Sergiy Matusevych wrote:
>>
>> Thank you so much Gon and the team!
>>
>> I got really stuck with this one last weekend. It looks like upgrading
>> other dependencies is trivial, except of Netty and Checkstyle/Findbugs.
>> It'd be great if you guys can help us with Netty while I take care of the
>> rest.
>>
>> Thank you,
>> Sergiy.
>>
>>
>> On Mon, Jan 27, 2020 at 3:04 PM Byung-Gon Chun <bg...@gmail.com> wrote:
>>
>>> Hi Sergiy,
>>>
>>> Thanks for the effort to upgrade Netty from 4.0 to 4.1.
>>> I'm adding Taegeon and Jangho, who know the codebase pretty well.
>>>
>>> Thanks.
>>> -Gon
>>>
>>> On Tue, Jan 28, 2020 at 5:22 AM Sergiy Matusevych <
>>> sergiy.matusevych@gmail.com> wrote:
>>>
>>>> Hi REEF team,
>>>>
>>>> I need your help investigating Wake unit test failures when using Netty
>>>> 4.1.
>>>>
>>>> Here's what's going on. I am updating REEF dependencies to make it more
>>>> in sync with Hadoop and Spark, among other things. One of the packages that
>>>> we use is Netty 4.0, and I want to switch it to version 4.1.42.Final.
>>>> (Besides, we already use Netty 4.1 in our gRPC bridge). Unfortunately,
>>>> drop-in replacement does not seem to work - there are two unit tests in
>>>> Wake that consistently fail under Netty 4.1.
>>>>
>>>> I've described the situation in this JIRA issue:
>>>> https://issues.apache.org/jira/browse/REEF-2059
>>>>
>>>> Looks like it has something to do with reconnect?
>>>>
>>>> Very likely, there are subtle changes in Netty behavior that we have to
>>>> take into account in our code. Can someone more familiar with Wake take a
>>>> look at the issue?
>>>>
>>>> To reproduce, do the following:
>>>>
>>>> 1) fork branch REEF-2057 from my github repo at
>>>> https://github.com/motus/reef/tree/REEF-2057
>>>>
>>>> 2) cd ./lang/java/reef-wake
>>>>
>>>> 3) run
>>>> mvn test -Dtest='org.apache.reef.wake.test.remote.RemoteManagerTest'
>>>>
>>>> Gon, maybe you or someone from your team can take a look at that?
>>>>
>>>> Meanwhile, I'll continue to upgrade other dependencies, and I hope we
>>>> can do a minor release when we're done.
>>>>
>>>> Cheers,
>>>> Sergiy.
>>>>
>>>>
>>>
>>> --
>>> Byung-Gon Chun
>>>
>>