You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2008/11/13 01:21:44 UTC
[jira] Created: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
NamenodeFsck creates a new DFSClient but never closes it
--------------------------------------------------------
Key: HADOOP-4647
URL: https://issues.apache.org/jira/browse/HADOOP-4647
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Tsz Wo (Nicholas), SZE
In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649853#action_12649853 ]
Hadoop QA commented on HADOOP-4647:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394205/4647_20081118.patch
against trunk revision 719651.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3622/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3622/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3622/console
This message is automatically generated.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647433#action_12647433 ]
Hairong Kuang commented on HADOOP-4647:
---------------------------------------
I prefer not to change the signature of lostFoundMove and then do the try-catch in lostFoundMove.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Attachment: 4647_20081118.patch
4647_20081118.patch: fixed the test setting a short blockreport interval.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649938#action_12649938 ]
Hudson commented on HADOOP-4647:
--------------------------------
Integrated in Hadoop-trunk #668 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/668/])
. NamenodeFsck should close the DFSClient it has created. (szetszwo)
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.3
>
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch, 4647_20081118_0.18.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648253#action_12648253 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4647:
------------------------------------------------
In build #3596, TestFsck failed at line 77: DFSTestUtil.waitReplication(...) during the test setup. Fsck was not involved yet. There are a lot of following messages in the log.
{noformat}
[junit] 2008-11-17 11:19:37,661 INFO FSNamesystem.audit (FSNamesystem.java:logAuditEvent(107)) - ugi=hudson,hudson
ip=/127.0.0.1 cmd=open src=/srcdat/57758981436956897 dst=null perm=null
[junit] File /srcdat/57758981436956897 has replication factor 4
[junit] Waiting for replication factor to drain
{noformat}
It seems that the file 57758981436956897 somehow has replication factor 4 and won't drain back to 3.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Attachment: 4647_20081113.patch
4647_20081113.patch: added try-finally for closing DFSClient.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.18.3
Status: Resolved (was: Patch Available)
I just committed this.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.3
>
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch, 4647_20081118_0.18.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648234#action_12648234 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4647:
------------------------------------------------
> Does this mean that the NameNode might leak a few socket descriptors every time an fsck is invoked?
This is probably the case. Also, it leak a thread each time in 0.17. All this leaking happen when running fsck with -move.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647447#action_12647447 ]
Hairong Kuang commented on HADOOP-4647:
---------------------------------------
+1
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
Status: Patch Available (was: Open)
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Attachment: 4647_20081118_0.18.patch
4647_20081118_0.18.patch: for 0.18
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch, 4647_20081118.patch, 4647_20081118_0.18.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648157#action_12648157 ]
Hadoop QA commented on HADOOP-4647:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393900/4647_20081113b.patch
against trunk revision 714107.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3596/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3596/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3596/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3596/console
This message is automatically generated.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Status: Open (was: Patch Available)
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4647) NamenodeFsck creates a new DFSClient
but never closes it
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4647:
-------------------------------------------
Attachment: 4647_20081113b.patch
4647_20081113b.patch: incorporated Hairong's comment.
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4647) NamenodeFsck creates a new
DFSClient but never closes it
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648103#action_12648103 ]
dhruba borthakur commented on HADOOP-4647:
------------------------------------------
Does this mean that the NameNode might leak a few socket descriptors every time an fsck is invoked?
> NamenodeFsck creates a new DFSClient but never closes it
> --------------------------------------------------------
>
> Key: HADOOP-4647
> URL: https://issues.apache.org/jira/browse/HADOOP-4647
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 4647_20081113.patch, 4647_20081113b.patch
>
>
> In NamenodeFsck.lostFoundMove(FileStatus file, LocatedBlocks blocks), a new DFSClient is created but never closed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.