You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2012/06/08 21:11:23 UTC
[jira] [Created] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Jason Lowe created HADOOP-8495:
----------------------------------
Summary: Update Netty to avoid leaking file descriptors during shuffle
Key: HADOOP-8495
URL: https://issues.apache.org/jira/browse/HADOOP-8495
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395935#comment-13395935 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2436 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2436/])
HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle (Jason Lowe via tgraves) (Revision 1351363)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated HADOOP-8495:
-------------------------------
Target Version/s: 0.23.3, 2.0.1-alpha
Status: Patch Available (was: Open)
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396784#comment-13396784 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1114 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1114/])
HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle (Jason Lowe via tgraves) (Revision 1351363)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396699#comment-13396699 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1081 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1081/])
HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle (Jason Lowe via tgraves) (Revision 1351363)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395982#comment-13395982 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2384 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2384/])
HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle (Jason Lowe via tgraves) (Revision 1351363)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated HADOOP-8495:
-------------------------------
Component/s: (was: io)
build
Affects Version/s: 3.0.0
2.0.0-alpha
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396734#comment-13396734 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #291 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/291/])
merge -r 1351363:1351364 from branch-2. FIXES: HADOOP-8495 (Revision 1351365)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351365
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HADOOP-8495.
---------------------------------
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated HADOOP-8495:
-------------------------------
Attachment: HADOOP-8495.patch
Patch to update the Netty dependency to 3.2.4.Final which has the fix for [NETTY-374|https://issues.jboss.org/browse/NETTY-374]. Manually tested with the issue described in MAPREDUCE-4298 to verify file descriptors are no longer leaked when I/O errors occur on the channel.
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395926#comment-13395926 ]
Thomas Graves commented on HADOOP-8495:
---------------------------------------
+1 lgtm.
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395938#comment-13395938 ]
Hudson commented on HADOOP-8495:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2364 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2364/])
HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle (Jason Lowe via tgraves) (Revision 1351363)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291962#comment-13291962 ]
Hadoop QA commented on HADOOP-8495:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531429/HADOOP-8495.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1100//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1100//console
This message is automatically generated.
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8495) Update Netty to avoid leaking file
descriptors during shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated HADOOP-8495:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.23.3
Status: Resolved (was: Patch Available)
Thanks Jason! I've committed this to trunk, branch-2, and branch-0.23.
> Update Netty to avoid leaking file descriptors during shuffle
> -------------------------------------------------------------
>
> Key: HADOOP-8495
> URL: https://issues.apache.org/jira/browse/HADOOP-8495
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 0.23.3
>
> Attachments: HADOOP-8495.patch
>
>
> Netty 3.2.3.Final has a known bug where writes to a closed channel do not have their futures invoked. See [Netty-374|https://issues.jboss.org/browse/NETTY-374]. This can lead to file descriptor leaks during shuffle as noted in MAPREDUCE-4298.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira