You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2008/03/21 21:33:24 UTC
[jira] Created: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
DFSInputStream 'pread' does not close its sockets
-------------------------------------------------
Key: HADOOP-3067
URL: https://issues.apache.org/jira/browse/HADOOP-3067
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Fix For: 0.17.0
{{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3067) DFSInputStream 'pread'
does not close its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581214#action_12581214 ]
rangadi edited comment on HADOOP-3067 at 3/21/08 4:07 PM:
---------------------------------------------------------------
Attached patch closes the sockets opened in {{fetchBlockByteRange()}}. Closing 'reader' is strictly not required but probably a good thing.
I think this patch uses IOUtils.closeStream() appropriately, let me know if you see other wise.
Regd unit test : TestCrcCorruption fails without this patch and passes with it when the the process is restricted to 1024 file descriptors. We could make hudson restrict the fds to 1024 as well. Otherwise it does not seem straight fwd to make sure all the sockets opened any where in Hadoop are closed when the test finishes..
was (Author: rangadi):
Attached patch closes the sockets opened in {{fetchBlockByteRange()}}. Closing 'reader' is strictly not required but probably a good thing.
I think this patch uses IOUtils.closeStream() appropriately, let me know it see other wise.
Regd unit test : TestCrcCorruption fails without this patch and passes with it when the the process is restricted to 1024 file descriptors. We could make hudson restrict the fds to 1024 as well. Otherwise it does not seem straight fwd to make sure all the sockets opened any where in Hadoop are closed when the test finishes..
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Environment: (was: DFSInputStream left sockets open. User Impact : positive but mostly unnoticed.)
Release Note: DFSInputStream left sockets open. User Impact : positive but mostly unnoticed.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Status: Patch Available (was: Open)
Thanks Hairong.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Description:
{{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
was:
{{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
Environment: DFSInputStream left sockets open. User Impact : positive but mostly unnoticed.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Environment: DFSInputStream left sockets open. User Impact : positive but mostly unnoticed.
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Attachment: HADOOP-3067.patch
Attached patch fixes one more bug that contributed to this problem :
{{BlockReader.checksumOk()}} is part of data read protocol and it should be inside BlockReader class. Earlier it was invoked by {{FSInputStream.read()}} only because BlockReader did not have access to the socket. Now the socket is stored. No API is changed. Since checksumOk() was invoked only by regular read and not by the position read used by unit test. This combined with the fact that the socket was not closed made the DataNodes wait on the socket.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3067) DFSInputStream 'pread' does not
close its sockets
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582513#action_12582513 ]
Hadoop QA commented on HADOOP-3067:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378509/HADOOP-3067.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2073/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2073/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2073/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2073/console
This message is automatically generated.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3067) DFSInputStream 'pread' does not
close its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582515#action_12582515 ]
Raghu Angadi commented on HADOOP-3067:
--------------------------------------
This is in fact triggered by a unit test. See the first comment above.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3067) DFSInputStream 'pread' does not
close its sockets
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582470#action_12582470 ]
Hairong Kuang commented on HADOOP-3067:
---------------------------------------
+1 This patch makes sure that the socket opened in fetchBlockByteRange is closed before the function returns.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Release Note: (was: DFSInputStream left sockets open. User Impact : positive but mostly unnoticed.)
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3067) DFSInputStream 'pread' does not
close its sockets
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582620#action_12582620 ]
Hudson commented on HADOOP-3067:
--------------------------------
Integrated in Hadoop-trunk #443 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/443/])
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch, HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3067) DFSInputStream 'pread' does not close
its sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3067:
---------------------------------
Attachment: HADOOP-3067.patch
Attached patch closes the sockets opened in {{fetchBlockByteRange()}}. Closing 'reader' is strictly not required but probably a good thing.
I think this patch uses IOUtils.closeStream() appropriately, let me know it see other wise.
Regd unit test : TestCrcCorruption fails without this patch and passes with it when the the process is restricted to 1024 file descriptors. We could make hudson restrict the fds to 1024 as well. Otherwise it does not seem straight fwd to make sure all the sockets opened any where in Hadoop are closed when the test finishes..
> DFSInputStream 'pread' does not close its sockets
> -------------------------------------------------
>
> Key: HADOOP-3067
> URL: https://issues.apache.org/jira/browse/HADOOP-3067
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-3067.patch
>
>
> {{DFSInputStream.read(int, buffer)}} does not close the sockets it opens. Main reason this problem did not show up till now is that pread interface is not used much.
> TestCrcCorruption failure first reported in HADOOP-2902 is caused by this bug. Hadoop 0.17 uses more file descriptors for each thread waiting on socket io. Since client does not close sockets, it leaves a lot of DataNode threads waiting in the unit test.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.