You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Sudheesh Katkam <sk...@mapr.com> on 2017/05/15 22:58:19 UTC
Upgrading Netty
Hi all,
As part of working on DRILL-5431 [1], I found a bug in Netty [2], which is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does anyone foresee issues with upgrading to the latest version of Netty? I noticed Apache Arrow upgraded to 4.0.41 [5].
Thank you,
Sudheesh
[1] https://issues.apache.org/jira/browse/DRILL-5431
[2] https://github.com/netty/netty/issues/6709
[3] https://github.com/netty/netty/pull/6713
[4] https://github.com/apache/drill/blob/master/pom.xml#L550
[5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3525c192745313b3da48
Re: Upgrading Netty
Posted by Kunal Khatua <kk...@mapr.com>.
Could this have been addressed in later versions of Netty?
Currently, initial performance numbers appear to have no impact on performance with the upgrade to version 4.0.48.
Scale and concurrency related tests, along with longevity are in progress.
~ Kunal
________________________________
From: Parth Chandra <pa...@apache.org>
Sent: Friday, May 19, 2017 2:29:50 PM
To: dev@drill.apache.org
Subject: Re: Upgrading Netty
Looks like the specific issue I was referring to was addressed in Netty
4.0.29 [1]. The comment for the commit is a little concerning:
<quote>
Result:
ThreadPoolCache is now also usable and so gives performance
improvements when allocation and deallocation thread are different.
Performance when using same thread for allocation and deallocation is
noticable worse then before.
</quote>
We might want to do a performance run to make sure things are no worse than
before.
[1]
https://github.com/netty/netty/commit/f765053ae740e300a6b696840d7dfe5de32afeb3
On Mon, May 15, 2017 at 5:46 PM, Parth Chandra <pa...@apache.org> wrote:
> The per thread allocation cache in Netty causes unbounded memory growth in
> Drill because we pass the ownership of a buffer from one thread to another.
> The version we use has a fix for the Drill use case where Netty will no
> longer add a buffer to its per thread cache if the buffer was allocated by
> a thread which is different from the thread freeing the buffer.
> This fix was reversed in a subsequent release and the latest version has
> the same issue.
> There might have been a fix in Netty for this in some other place which I
> am not aware of (perhaps they removed it altogether as Paul seems to have
> seen).
> AFAIK, we do not have a direct reference to that code in Drill's
> allocator. If you try to upgrade and hit an issue, post it here.
> If you are able to upgrade the Netty version, then run a longevity test to
> make sure there is no 'leaking' of memory from one thread to another.
>
>
>
> On Mon, May 15, 2017 at 4:08 PM, Paul Rogers <pr...@mapr.com> wrote:
>
>> As it turns out, Drill makes clever use of the internal details of the
>> Netty memory allocator. But, that code changed significantly in the last
>> couple of years. When I attempted to upgrade, I found that the private
>> features of Netty that the Drill allocator uses no longer exist in the
>> latest Netty.
>>
>> So, someone will need to understand what that part of the Drill allocator
>> does and design an alternative integration.
>>
>> The particular issue seems to be that Netty had a per-thread allocation
>> cache which seems to not exist in the latest version.
>>
>> - Paul
>>
>> > On May 15, 2017, at 3:58 PM, Sudheesh Katkam <sk...@mapr.com> wrote:
>> >
>> > Hi all,
>> >
>> > As part of working on DRILL-5431 [1], I found a bug in Netty [2], which
>> is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does
>> anyone foresee issues with upgrading to the latest version of Netty? I
>> noticed Apache Arrow upgraded to 4.0.41 [5].
>> >
>> > Thank you,
>> > Sudheesh
>> >
>> > [1] https://issues.apache.org/jira/browse/DRILL-5431
>> > [2] https://github.com/netty/netty/issues/6709
>> > [3] https://github.com/netty/netty/pull/6713
>> > [4] https://github.com/apache/drill/blob/master/pom.xml#L550
>> > [5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3
>> 525c192745313b3da48
>>
>>
>
Re: Upgrading Netty
Posted by Parth Chandra <pa...@apache.org>.
Looks like the specific issue I was referring to was addressed in Netty
4.0.29 [1]. The comment for the commit is a little concerning:
<quote>
Result:
ThreadPoolCache is now also usable and so gives performance
improvements when allocation and deallocation thread are different.
Performance when using same thread for allocation and deallocation is
noticable worse then before.
</quote>
We might want to do a performance run to make sure things are no worse than
before.
[1]
https://github.com/netty/netty/commit/f765053ae740e300a6b696840d7dfe5de32afeb3
On Mon, May 15, 2017 at 5:46 PM, Parth Chandra <pa...@apache.org> wrote:
> The per thread allocation cache in Netty causes unbounded memory growth in
> Drill because we pass the ownership of a buffer from one thread to another.
> The version we use has a fix for the Drill use case where Netty will no
> longer add a buffer to its per thread cache if the buffer was allocated by
> a thread which is different from the thread freeing the buffer.
> This fix was reversed in a subsequent release and the latest version has
> the same issue.
> There might have been a fix in Netty for this in some other place which I
> am not aware of (perhaps they removed it altogether as Paul seems to have
> seen).
> AFAIK, we do not have a direct reference to that code in Drill's
> allocator. If you try to upgrade and hit an issue, post it here.
> If you are able to upgrade the Netty version, then run a longevity test to
> make sure there is no 'leaking' of memory from one thread to another.
>
>
>
> On Mon, May 15, 2017 at 4:08 PM, Paul Rogers <pr...@mapr.com> wrote:
>
>> As it turns out, Drill makes clever use of the internal details of the
>> Netty memory allocator. But, that code changed significantly in the last
>> couple of years. When I attempted to upgrade, I found that the private
>> features of Netty that the Drill allocator uses no longer exist in the
>> latest Netty.
>>
>> So, someone will need to understand what that part of the Drill allocator
>> does and design an alternative integration.
>>
>> The particular issue seems to be that Netty had a per-thread allocation
>> cache which seems to not exist in the latest version.
>>
>> - Paul
>>
>> > On May 15, 2017, at 3:58 PM, Sudheesh Katkam <sk...@mapr.com> wrote:
>> >
>> > Hi all,
>> >
>> > As part of working on DRILL-5431 [1], I found a bug in Netty [2], which
>> is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does
>> anyone foresee issues with upgrading to the latest version of Netty? I
>> noticed Apache Arrow upgraded to 4.0.41 [5].
>> >
>> > Thank you,
>> > Sudheesh
>> >
>> > [1] https://issues.apache.org/jira/browse/DRILL-5431
>> > [2] https://github.com/netty/netty/issues/6709
>> > [3] https://github.com/netty/netty/pull/6713
>> > [4] https://github.com/apache/drill/blob/master/pom.xml#L550
>> > [5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3
>> 525c192745313b3da48
>>
>>
>
Re: Upgrading Netty
Posted by Parth Chandra <pa...@apache.org>.
The per thread allocation cache in Netty causes unbounded memory growth in
Drill because we pass the ownership of a buffer from one thread to another.
The version we use has a fix for the Drill use case where Netty will no
longer add a buffer to its per thread cache if the buffer was allocated by
a thread which is different from the thread freeing the buffer.
This fix was reversed in a subsequent release and the latest version has
the same issue.
There might have been a fix in Netty for this in some other place which I
am not aware of (perhaps they removed it altogether as Paul seems to have
seen).
AFAIK, we do not have a direct reference to that code in Drill's allocator.
If you try to upgrade and hit an issue, post it here.
If you are able to upgrade the Netty version, then run a longevity test to
make sure there is no 'leaking' of memory from one thread to another.
On Mon, May 15, 2017 at 4:08 PM, Paul Rogers <pr...@mapr.com> wrote:
> As it turns out, Drill makes clever use of the internal details of the
> Netty memory allocator. But, that code changed significantly in the last
> couple of years. When I attempted to upgrade, I found that the private
> features of Netty that the Drill allocator uses no longer exist in the
> latest Netty.
>
> So, someone will need to understand what that part of the Drill allocator
> does and design an alternative integration.
>
> The particular issue seems to be that Netty had a per-thread allocation
> cache which seems to not exist in the latest version.
>
> - Paul
>
> > On May 15, 2017, at 3:58 PM, Sudheesh Katkam <sk...@mapr.com> wrote:
> >
> > Hi all,
> >
> > As part of working on DRILL-5431 [1], I found a bug in Netty [2], which
> is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does
> anyone foresee issues with upgrading to the latest version of Netty? I
> noticed Apache Arrow upgraded to 4.0.41 [5].
> >
> > Thank you,
> > Sudheesh
> >
> > [1] https://issues.apache.org/jira/browse/DRILL-5431
> > [2] https://github.com/netty/netty/issues/6709
> > [3] https://github.com/netty/netty/pull/6713
> > [4] https://github.com/apache/drill/blob/master/pom.xml#L550
> > [5] https://github.com/apache/arrow/commit/
> 3487c2f0cdc2297a80ba3525c192745313b3da48
>
>
Re: Upgrading Netty
Posted by Paul Rogers <pr...@mapr.com>.
As it turns out, Drill makes clever use of the internal details of the Netty memory allocator. But, that code changed significantly in the last couple of years. When I attempted to upgrade, I found that the private features of Netty that the Drill allocator uses no longer exist in the latest Netty.
So, someone will need to understand what that part of the Drill allocator does and design an alternative integration.
The particular issue seems to be that Netty had a per-thread allocation cache which seems to not exist in the latest version.
- Paul
> On May 15, 2017, at 3:58 PM, Sudheesh Katkam <sk...@mapr.com> wrote:
>
> Hi all,
>
> As part of working on DRILL-5431 [1], I found a bug in Netty [2], which is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does anyone foresee issues with upgrading to the latest version of Netty? I noticed Apache Arrow upgraded to 4.0.41 [5].
>
> Thank you,
> Sudheesh
>
> [1] https://issues.apache.org/jira/browse/DRILL-5431
> [2] https://github.com/netty/netty/issues/6709
> [3] https://github.com/netty/netty/pull/6713
> [4] https://github.com/apache/drill/blob/master/pom.xml#L550
> [5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3525c192745313b3da48