You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2012/01/19 19:42:41 UTC
[jira] [Created] (HDFS-2810) Leases not properly getting renewed by
clients
Leases not properly getting renewed by clients
----------------------------------------------
Key: HDFS-2810
URL: https://issues.apache.org/jira/browse/HDFS-2810
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.23.0, 0.24.0
Reporter: Todd Lipcon
Priority: Critical
We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189665#comment-13189665 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1633 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1633/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233794
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2810) Leases not properly getting renewed by
clients
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2810:
------------------------------
Attachment: hdfs-2810.txt
I think the attached patch fixes the issue. Will verify on a cluster as well.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2810) Leases not properly getting renewed by
clients
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2810:
------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
My HBase cluster remains up a few hours later. Committed to trunk and 23, thanks for the reviews.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189596#comment-13189596 ]
Todd Lipcon commented on HDFS-2810:
-----------------------------------
Letting my test cluster run for a couple more hours, then will commit it. Thanks for the reviews.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189800#comment-13189800 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #964 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/964/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233794
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2810) Leases not properly getting renewed by
clients
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2810:
------------------------------
Status: Patch Available (was: Open)
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189377#comment-13189377 ]
Todd Lipcon commented on HDFS-2810:
-----------------------------------
I think I see the bug here. DFSClient.renewLease has:
{code}
void renewLease() throws IOException {
if (clientRunning && !isFilesBeingWrittenEmpty()) {
namenode.renewLease(clientName);
}
}
{code}
and LeaseRenewer only calls {{renewLease()}} on the first DFSClient with any given name.
So, if there are multiple DFSClients with the same clientName, but the first of them has no files open, then the lease renewer will stop actually renewing the lease.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189453#comment-13189453 ]
Hadoop QA commented on HDFS-2810:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511172/hdfs-2810.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 21 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestLeaseRenewer
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1784//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1784//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1784//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1784//console
This message is automatically generated.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2810) Leases not properly getting renewed by
clients
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2810:
------------------------------
Attachment: hdfs-2810.txt
Oops, the bumping of the logging level caused the thread name to be more verbose, which broke testThreadName. Reverting that part of the change.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189781#comment-13189781 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #144 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/144/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233793
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189677#comment-13189677 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #405 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/405/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233793
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189669#comment-13189669 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1560 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1560/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233794
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Todd Lipcon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned HDFS-2810:
---------------------------------
Assignee: Todd Lipcon
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189793#comment-13189793 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #166 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/166/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233793
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189689#comment-13189689 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1578 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1578/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233794
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189519#comment-13189519 ]
Hadoop QA commented on HDFS-2810:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511187/hdfs-2810.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 21 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1785//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1785//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1785//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1785//console
This message is automatically generated.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189664#comment-13189664 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #382 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/382/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233793
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189486#comment-13189486 ]
Eli Collins commented on HDFS-2810:
-----------------------------------
+1 lgtm, nice find.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189589#comment-13189589 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2810:
----------------------------------------------
Good catch! Thanks for fixing this.
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189671#comment-13189671 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Common-0.23-Commit #392 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/392/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233793
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2810) Leases not properly getting renewed
by clients
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189770#comment-13189770 ]
Hudson commented on HDFS-2810:
------------------------------
Integrated in Hadoop-Hdfs-trunk #931 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/931/])
HDFS-2810. Leases not getting renewed properly by clients. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1233794
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
> Leases not properly getting renewed by clients
> ----------------------------------------------
>
> Key: HDFS-2810
> URL: https://issues.apache.org/jira/browse/HDFS-2810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2810.txt, hdfs-2810.txt
>
>
> We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira