You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2011/04/27 06:31:03 UTC
[jira] [Created] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Share LeaseChecker thread among DFSClients
------------------------------------------
Key: HDFS-1865
URL: https://issues.apache.org/jira/browse/HDFS-1865
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026865#comment-13026865 ]
Devaraj Das commented on HDFS-1865:
-----------------------------------
HDFS-1131 is one use case (maybe that jira can be closed as a duplicate of this one).
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030907#comment-13030907 ]
Suresh Srinivas commented on HDFS-1865:
---------------------------------------
Minor comments:
# I prefer to retain using "DFSClient" instead of simple class name while building the client name.
# TestLease could be simplified using mockito
# Javadoc could be more descriptive for LeaseRenewer
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Attachment: h1865_20110504.patch
h1865_20110504.patch: updated with trunk.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks Suresh for the reviews.
I have committed this.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0
>
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Attachment: h1865_20110503.patch
h1865_20110503.patch: 1st patch, need some new tests.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030976#comment-13030976 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1865:
----------------------------------------------
The failure {{TestUnderReplicatedBlocks}} was caused by timeout; see HDFS-342. I can reproduce the failure in my machine without the patch.
The failure {{TestFiDataTransferProtocol2}} was caused by NullPointerException in ipc.Server; see HADOOP-7270.
The other failed tests are known.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030975#comment-13030975 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1865:
----------------------------------------------
The failure {{TestUnderReplicatedBlocks}} was caused by timeout; see HDFS-342. I can reproduce the failure in my machine without the patch.
The failure {{TestFiDataTransferProtocol2}} was caused by NullPointerException in ipc.Server; see HADOOP-7270.
The other failed tests are known.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Status: Patch Available (was: Open)
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030560#comment-13030560 ]
Hadoop QA commented on HDFS-1865:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12478552/h1865_20110508.patch
against trunk revision 1100811.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestFileConcurrentReader
org.apache.hadoop.hdfs.TestReadWhileWriting
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/467//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/467//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/467//console
This message is automatically generated.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Attachment: h1865_20110508b.patch
h1865_20110508b.patch: fixed {{TestReadWhileWriting}}. The other failed tests are not related.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026860#comment-13026860 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1865:
----------------------------------------------
I am actually thinking to have one thread per namenode per user. In such case, could we use the same client name for all {{DFSClient}} instances?
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030584#comment-13030584 ]
Hadoop QA commented on HDFS-1865:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12478562/h1865_20110508b.patch
against trunk revision 1100811.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/468//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/468//console
This message is automatically generated.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030950#comment-13030950 ]
Hadoop QA commented on HDFS-1865:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12478639/h1865_20110509.patch
against trunk revision 1101137.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestUnderReplicatedBlocks
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/469//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/469//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/469//console
This message is automatically generated.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026643#comment-13026643 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1865:
----------------------------------------------
I have considered share {{DFSClient}} directly. However, {{DFSClient}} depends on conf parameters like timeout, packet size, etc. So it cannot be shared.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026857#comment-13026857 ]
dhruba borthakur commented on HDFS-1865:
----------------------------------------
hi nicholas, I do not undertand your question/proposal. A single JVM can have multiple DFSClients, each talking to different namenodes. are you proposing that there be one thread to renew lease to all the namenodes?
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030927#comment-13030927 ]
Suresh Srinivas commented on HDFS-1865:
---------------------------------------
+1 for the patch.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Attachment: h1865_20110508.patch
h1865_20110508.patch: added a test and incorporated Suresh's comments.
> Why do you check {if (r == stored) } in Factory#remove()?
Since a renewer may expire, the stored renewer can be different. Added a comment in the code.
> Why did we choose to use volatile for sleepPeriod and synchronized for gracePeriod and other periods?
Used synchronized for both now.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026883#comment-13026883 ]
dhruba borthakur commented on HDFS-1865:
----------------------------------------
Ok, got it.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030151#comment-13030151 ]
Suresh Srinivas commented on HDFS-1865:
---------------------------------------
Comments:
# DFSClient.java
#* DFSClient#filesBeingWritten - worth adding a comment that a file can be opened for write by a DFSClient only once.
# LeaseRenewer.java
#* In the class document, it is good to describe lease mechanism briefly. This would help people understand the code better. Adding additional comments on how LeaseRenewer works would also be good. Adding information about synchronization will also help understand the code better.
#* Please throw HadoopIllegalArgumentException instead of NullPointerException.
#* Please do not access DFSClient#filesBeingWritten in this class. Add methods to DFSClient to do this.
#* Factory#remove - please avoid adding synchronized on renewer. Instead call a method on renewer, that is synchronized.
#* Minor: Simplify the code in in LeaseRenewer#get(). If map returns empty then add it to map and return it. You do not not need if else. You can do it with just if and can get rid of "r".
#* Copy on array list for DFSClients as an alternative to making a copy every time in renew()?
#* Why do you check {if (r == stored) } in Factory#remove()?
#* Why did we choose to use volatile for sleepPeriod and synchronized for gracePeriod and other periods?
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031912#comment-13031912 ]
Hudson commented on HDFS-1865:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #637 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/637/])
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0
>
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026861#comment-13026861 ]
dhruba borthakur commented on HDFS-1865:
----------------------------------------
Oh ok, can you also explain the usecase why you have multiple instances of DFSClient inside the same jvm (aren't all config properties for file creation already part of the FileSystem.create api?)
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026131#comment-13026131 ]
dhruba borthakur commented on HDFS-1865:
----------------------------------------
why not share the DFSClient itself?
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026645#comment-13026645 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1865:
----------------------------------------------
Dhruba, question to you: {{DFSClient}} has a unique client name even there are multiple {{DFSClient}} instances in a JVM. Do you think that we can use the same name for all {{DFSClient}} instances?
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037353#comment-13037353 ]
Hudson commented on HDFS-1865:
------------------------------
Integrated in Hadoop-Hdfs-trunk #673 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/])
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0
>
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1865) Share LeaseChecker thread among
DFSClients
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1865:
-----------------------------------------
Attachment: h1865_20110509.patch
h1865_20110509.patch: incorporated the minor comments.
> Share LeaseChecker thread among DFSClients
> ------------------------------------------
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1865_20110503.patch, h1865_20110504.patch, h1865_20110508.patch, h1865_20110508b.patch, h1865_20110509.patch
>
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira