You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2021/04/23 14:58:00 UTC
[jira] [Comment Edited] (TEZ-4157) ShuffleHandler: upgrade to
Netty4 and remove Netty3 dependency from tez
[ https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330838#comment-17330838 ]
László Bodor edited comment on TEZ-4157 at 4/23/21, 2:57 PM:
-------------------------------------------------------------
[~jeagles]: it was an offline discussion with [~ashutoshc] regarding the code and testing results, so this PR doesn't contain that
however, there were some comments on the corresponding hive change: https://github.com/apache/hive/pull/1778
also, I opened a netty issue, when I suspected a netty bug, but turned out I was using it improperly: https://github.com/netty/netty/issues/11155
all the fixes are included into this patch
sorry for the confusion
was (Author: abstractdog):
[~jeagles]: it was an offline discussion with [~ashutoshc] regarding the code and testing results, so this PR doesn't contain that
however, there were some comments on the corresponding hive change: https://github.com/apache/hive/pull/1778
also, I opened a netty issue, when I suspected a netty bug, but turned out I was using it improperly: https://github.com/netty/netty/issues/11155
all the fixes are included into this patch
> ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez
> -----------------------------------------------------------------------
>
> Key: TEZ-4157
> URL: https://issues.apache.org/jira/browse/TEZ-4157
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Fix For: 0.10.1, 0.9.3
>
> Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch, TEZ-4157.03.patch, TEZ-4157.04.patch, TEZ-4157.05.patch, TEZ-4157.06.patch, dep.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> -In the dependency tree, there are 2 occurrences of compile scope direct netty dependencies, however, they're not used at all. I compiled locally successfully without them. E.g. when investigating blackduck alerts (complaining about netty deps for current 3.10.5.Final), it would be cleaner to start from a dependency tree where Tez doesn't depend on netty directly in order to eliminate its responsibility (and move the focus to underlying hadoop for instance).-
> Tez depends on netty3 almost only in ShuffleHandler and some related classes. We can eliminate netty3 by upgrading it, but this effort might involve some testing due to fundamental [changes from netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we don't have a reference yet, as [hadoop's ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java] is still on netty3.
> As per the netty documentation, we can also expect some performance improvement (e.g. Pooled buffers).
> Background:
> netty4 migration guideline: https://netty.io/wiki/new-and-noteworthy-in-4.0.html
> articles of possible performance improvement:
> https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html
> https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)