You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Uma Maheswara Rao G (Created) (JIRA)" <ji...@apache.org> on 2011/10/23 18:30:32 UTC
[jira] [Created] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
[webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
----------------------------------------------------------------------------------------------------
Key: HDFS-2494
URL: https://issues.apache.org/jira/browse/HDFS-2494
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.24.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
As part of the reliable test,
Scenario:
Initially check the socket count. ---there are aroud 42 sockets are there.
open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
Here is the netstat result:
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
This count is coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136601#comment-13136601 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #62 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/62/])
svn merge -c 1189360 from trunk for HDFS-2494.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189520
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136044#comment-13136044 ]
Hadoop QA commented on HDFS-2494:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500889/HDFS-2494-20.205Security.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1460//console
This message is automatically generated.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134707#comment-13134707 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
Ok Nicholas,
Since this JIRA raised for OPEN, i thought file separate JIRA for put,post. Now i included them in this JIRA.
I think we can the edit title of this JIRA as well.
Thanks
Uma
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Attachment: h2494_20111024.patch
Hi Uma, thanks for catching the bug. The input stream is not closed in the case {{n == null}}. The two {{IOUtils.copyBytes(..}}}s behave differently: the second one closes in but the first does not.
h2494_20111024.patch: close in.
Could you test the patch?
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: h2494_20111024.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137107#comment-13137107 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #64 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/])
svn merge -c 1189360 from trunk for HDFS-2494.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189520
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136603#comment-13136603 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Common-0.23-Commit #66 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/66/])
svn merge -c 1189360 from trunk for HDFS-2494.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189520
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HDFS-2316
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Target Version/s: 0.20.205.1, 0.24.0 (was: 0.24.0, 0.20.205.1)
Fix Version/s: 0.23.0
Merged to 0.23.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136605#comment-13136605 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #67 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/67/])
svn merge -c 1189360 from trunk for HDFS-2494.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189520
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2494:
--------------------------------------
Attachment: HDFS-2494.patch
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Attachment: h2494_20111024b.patch
> since we are passing the closed flag as false, this will not close ...
You are right. Here is a new patch.
h2494_20111024b.patch: close for both cases.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133865#comment-13133865 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
In DatanodeWebHdfsMethods.java,
Looks we are using the DFSInputStream in StreamingOutput#write method
{code}
final StreamingOutput streaming = new StreamingOutput() {
@Override
public void write(final OutputStream out) throws IOException {
final Long n = length.getValue();
if (n == null) {
IOUtils.copyBytes(in, out, b);
} else {
IOUtils.copyBytes(in, out, n, false);
}
}
};
{code}
Here we dont have the handle to clean them and the invokation of this write API will come from the provider latter. So, we can not clean it before return in get api.
Since the complete content flush is happening from IOUtils.copyBytes, one work around would be to clen them in write method itself. But i am not sure this approach is correct in this case.
Thanks
Uma
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Arpit Gupta (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arpit Gupta updated HDFS-2494:
------------------------------
Target Version/s: 0.20.205.1, 0.24.0 (was: 0.24.0)
we should fix this for 205.1 also
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137137#comment-13137137 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Hdfs-trunk #845 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/845/])
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189360
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136249#comment-13136249 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1174 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1174/])
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189360
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134626#comment-13134626 ]
Hadoop QA commented on HDFS-2494:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500539/HDFS-2494.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestFileCreationClient
org.apache.hadoop.hdfs.TestSetrepIncreasing
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1433//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1433//console
This message is automatically generated.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136203#comment-13136203 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1160 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1160/])
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189360
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134745#comment-13134745 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2494:
----------------------------------------------
+1 patch looks good. Thanks a lot for fixing this.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135583#comment-13135583 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2494:
----------------------------------------------
Uma, could you post a patch for 0.20-security?
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137042#comment-13137042 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #52 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/])
svn merge -c 1189360 from trunk for HDFS-2494.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189520
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135549#comment-13135549 ]
Hadoop QA commented on HDFS-2494:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500598/HDFS-2494.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.TestBalancerBandwidth
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1443//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1443//console
This message is automatically generated.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2494:
--------------------------------------
Attachment: HDFS-2494-20.205Security.patch
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136197#comment-13136197 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1237 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1237/])
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189360
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134345#comment-13134345 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
Hi Nicholas,
Thanks for the patch.
I just checked the patch!.
When the below IOUtils.copyBytes will close the stream.
{code}
} else {
IOUtils.copyBytes(in, out, n, false);
}
{code}
since we are passing the closed flag as false, this will not close the streams in copybytes methods. So, when this will be closed? Am i missing something here?
I will verify the patch in my env and post the results.
Thanks
Uma
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: h2494_20111024.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2494:
----------------------------------
Target Version/s: 1.0.0, 0.24.0 (was: 0.24.0, 1.0.0)
Fix Version/s: (was: 1.1.0)
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: webhdfs
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.23.0, 0.24.0, 1.0.0
>
> Attachments: h2494_20111024b.patch, h2494_20111024.patch, HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134476#comment-13134476 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
Hi Nicholas,
I just verified the patch with my environment, looks there are more sockets still be leaking.
I just gave the fix by closing dfsclient as well.
After this step, sockets came down,
C:\Users\uma>netstat | grep ESTABLISHED|wc -l
50
C:\Users\uma>netstat | grep ESTABLISHED|wc -l
58
C:\Users\uma>netstat | grep ESTABLISHED|wc -l
58
I uploaded the patch with it. Can you please take a look?
Thanks
Uma
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2494:
--------------------------------------
Description:
As part of the reliable test,
Scenario:
Initially check the socket count. ---there are aroud 42 sockets are there.
open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
Here is the netstat result:
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
This count is not coming down.
was:
As part of the reliable test,
Scenario:
Initially check the socket count. ---there are aroud 42 sockets are there.
open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
Here is the netstat result:
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
2042
This count is coming down.
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Resolution: Fixed
Fix Version/s: 0.24.0
0.20.206.0
0.20.205.1
Target Version/s: 0.20.205.1, 0.24.0 (was: 0.24.0, 0.20.205.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have committed this. Thanks a lot, Uma!
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136051#comment-13136051 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
Patch is for 20.205 security branch!
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134557#comment-13134557 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2494:
----------------------------------------------
> I just gave the fix by closing dfsclient as well.
If it is the case, the dfsclient in put(..) and post(..) should also be closed. Could you also change them?
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137067#comment-13137067 ]
Hudson commented on HDFS-2494:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #873 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/873/])
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. Contributed by Uma Maheswara Rao G
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189360
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: HDFS-2494-20.205Security.patch, HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2494) [webhdfs] When Getting the file
using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135906#comment-13135906 ]
Uma Maheswara Rao G commented on HDFS-2494:
-------------------------------------------
Sorry Nicholas, i was in Travel. So, i could not post the patch yesterday.
I will post the patch for security, in couple of minutes.
Thanks
Uma
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2494:
-----------------------------------------
Status: Patch Available (was: Open)
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2494:
--------------------------------------
Attachment: HDFS-2494.patch
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2494) [webhdfs] When Getting the file using
OP=OPEN with DN http address, ESTABLISHED sockets are growing.
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2494:
--------------------------------------
Affects Version/s: 0.23.0
0.20.205.0
> [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-2494
> URL: https://issues.apache.org/jira/browse/HDFS-2494
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.205.0, 0.23.0, 0.24.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2494.patch, HDFS-2494.patch, h2494_20111024.patch, h2494_20111024b.patch
>
>
> As part of the reliable test,
> Scenario:
> Initially check the socket count. ---there are aroud 42 sockets are there.
> open the file with DataNode http address using op=OPEN request parameter about 500 times in loop.
> Wait for some time and check the socket count. --- There are thousands of ESTABLISHED sockets are growing. ~2052
> Here is the netstat result:
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> C:\Users\uma>netstat | grep 127.0.0.1 | grep ESTABLISHED |wc -l
> 2042
> This count is not coming down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira