You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2008/03/21 23:37:24 UTC
[jira] Created: (HADOOP-3071) DFSOutputStream does not close all
the sockets
DFSOutputStream does not close all the sockets
----------------------------------------------
Key: HADOOP-3071
URL: https://issues.apache.org/jira/browse/HADOOP-3071
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.16.0
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Blocker
Fix For: 0.16.2
When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3071:
---------------------------------
Fix Version/s: (was: 0.16.2)
0.17.0
Priority: Major (was: Blocker)
Description:
When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
was:
When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
Affects Version/s: (was: 0.16.0)
0.17.0
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581230#action_12581230 ]
Raghu Angadi commented on HADOOP-3071:
--------------------------------------
This does not affect 0.16. Closing s.getOutputStream() also closes the socket (not just the otherway round). o.a.h.SocketOutputStream() should do the same. But still I think patch is good as it is.. it cleans up the code a bit and closes the socket if connection fails inside createBlockOutputStream().
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3071:
---------------------------------
Attachment: HADOOP-3071.patch
Attached patch adds {{closeBlockStreams()}}.
Without this patch there around 200 sockets to DataNodes in CLOSE_WAIT at the end of TestCrcCorruption unit test. With the patch there are no sockets open to DataNodes.
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.16.2
>
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581795#action_12581795 ]
dhruba borthakur commented on HADOOP-3071:
------------------------------------------
Code looks good. One question: the original code closes blockStream first and then close blockReplyStream. The new code closes blockReplyStream first. Is this on purpose?
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3071:
------------------------------------
Component/s: dfs
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582351#action_12582351 ]
Raghu Angadi commented on HADOOP-3071:
--------------------------------------
Thanks Dhruba. The order does not matter. In my earlier changes, I had moved setting blockStream to null to the end and later forgot to move it back. I can change the order.
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3071) DFSOutputStream does not close all
the sockets
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3071:
------------------------------------
Fix Version/s: (was: 0.17.0)
> DFSOutputStream does not close all the sockets
> ----------------------------------------------
>
> Key: HADOOP-3071
> URL: https://issues.apache.org/jira/browse/HADOOP-3071
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: HADOOP-3071.patch
>
>
> When DFSOutputStream writes to multiple blocks, it closes only the socket opened for the last block. When it is done with writing to one block it should close the socket.
> I noticed this when I was fixing HADOOP-3067. After fixing HADOOP-3067, there were still a lot of sockets open (but not enough to fail the tests). These sockets were used to write to blocks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.