You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (Created) (JIRA)" <ji...@apache.org> on 2011/11/23 00:54:39 UTC
[jira] [Created] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
HBASE-4789 does overzealous pruning of seqids
---------------------------------------------
Key: HBASE-4853
URL: https://issues.apache.org/jira/browse/HBASE-4853
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
At a minimum, these lines need removing:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
index 623edbe..a0bbe01 100644
--- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
+++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
@@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
// Cleaning up of lastSeqWritten is in the finally clause because we
// don't want to confuse getOldestOutstandingSeqNum()
this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
- Long l = this.lastSeqWritten.remove(encodedRegionName);
- if (l != null) {
- LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
- Bytes.toString(encodedRegionName) + ", seqid=" + l);
- }
this.cacheFlushLock.unlock();
}
}
{code}
... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v8.txt
The odd failing of TestGlobalMemStoreSize was edits going into meta while other regions were flushing. Moved things around so we flush meta last. That should do it.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155733#comment-13155733 ]
stack commented on HBASE-4853:
------------------------------
Testing, this patch does not fix the issue of our not cleaning up WALs. I'm digging still.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156583#comment-13156583 ]
Hudson commented on HBASE-4853:
-------------------------------
Integrated in HBase-0.92-security #12 (See [https://builds.apache.org/job/HBase-0.92-security/12/])
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156524#comment-13156524 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504984/4853-v10.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/361//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v9.txt
Same patch. Trigger another build.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v7.txt
v7 still needs work; i'd like to nail TestGlobalMemStore before putting it up for review (worst comes to worst, I'll take Ted's trick)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156534#comment-13156534 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504960/4853-v9.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -162 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestInstantSchemaChange
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/359//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/359//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/359//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156366#comment-13156366 ]
stack commented on HBASE-4853:
------------------------------
hmm... that don't fail for me and the change shouldn't effect this test.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156283#comment-13156283 ]
stack commented on HBASE-4853:
------------------------------
Looks like this commit by me broke our memstore sizing: HBASE-4722. It takes memstore flush size outside of an update lock (more edits may have come in in meantime).
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155605#comment-13155605 ]
Jean-Daniel Cryans commented on HBASE-4853:
-------------------------------------------
+1
I'm still confused on how we end up with an id in there even tho we also remove it at the beginning of startCacheFlush.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156489#comment-13156489 ]
Ted Yu commented on HBASE-4853:
-------------------------------
Patch v9 is good.
Minor comment:
{code}
// it could get edits while other flushes are running since this a
{code}
I think the tail should read 'since this is a'
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156332#comment-13156332 ]
stack commented on HBASE-4853:
------------------------------
Here's some explaination:
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
On flush of memstores, we were decrementing the global region
memory size by the size of the global memstore AT THE TIME OF
THE DECREMENT rather than decrementing by the flush size (some
edits may very well have come in in between the setup of flush
and decrement time). This change undoes a brain-dead change
of mine in hbase-4722. That broke this.
M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
Remove flagging of the original problem, our leaving an old
edit id in the lastSeqWritten for a region that was offline.
I tried to write a test but its too tough at mo. You need to get some edits into the memstore AFTER the update lock is freed down in internalFlushCache but BEFORE we decrement memstore size. Only way to make it work would be by mod'ing HRegion to insert a do-nothing method. Too dumb.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155607#comment-13155607 ]
Todd Lipcon commented on HBASE-4853:
------------------------------------
Why are our tests not freaking out with this bug?
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156430#comment-13156430 ]
Ted Yu commented on HBASE-4853:
-------------------------------
With patch v8, I got the following after 4 iterations:
{code}
testGlobalMemStore(org.apache.hadoop.hbase.TestGlobalMemStoreSize) Time elapsed: 9.521 sec <<< FAILURE!
java.lang.AssertionError: Server=10.246.204.31,53367,1322094489131, i=2 expected:<0> but was:<608>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.hadoop.hbase.TestGlobalMemStoreSize.testGlobalMemStore(TestGlobalMemStoreSize.java:128)
{code}
Here is tail of test output:
{code}
2011-11-23 16:28:15,599 INFO [main] regionserver.StoreFile$Reader(1455): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d080dcfc9b854237b0e8bd884f1e395b
2011-11-23 16:28:15,600 DEBUG [main] regionserver.Store(610): Renaming flushed file at hdfs://localhost:53343/user/zhihyu/.META./1028785192/.tmp/d080dcfc9b854237b0e8bd884f1e395b to hdfs://localhost:53343/user/zhihyu/.META./1028785192/info/d080dcfc9b854237b0e8bd884f1e395b
2011-11-23 16:28:15,606 INFO [main] regionserver.StoreFile$Reader(1455): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d080dcfc9b854237b0e8bd884f1e395b
2011-11-23 16:28:15,606 INFO [main] regionserver.Store(631): Added hdfs://localhost:53343/user/zhihyu/.META./1028785192/info/d080dcfc9b854237b0e8bd884f1e395b, entries=47, sequenceid=36, filesize=8.4k
2011-11-23 16:28:15,607 INFO [main] regionserver.HRegion(1394): Finished memstore flush of ~16.6k/17000 for region .META.,,1.1028785192 in 437ms, sequenceid=36, compaction requested=false
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(142): Flush .META.,,1.1028785192 on 10.246.204.31,53367,1322094489131, false, size=608
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(112): Post flush on 10.246.204.31,53367,1322094489131
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(125): .META.,,1.1028785192 608
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(125): TestGlobalMemStoreSize,aaaaa,1322094494548.21d8ad184c650c0be744b624a11078c1. 0
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(125): TestGlobalMemStoreSize,f\xC2/\xE6\xBF,1322094494555.9db9aa6f0a4f29d6f449209c319cd77a. 0
2011-11-23 16:28:15,607 INFO [main] hbase.TestGlobalMemStoreSize(125): TestGlobalMemStoreSize,l"\xFEl\x1D,1322094494561.0182399055dc627bb152408d8d2234dd. 0
2011-11-23 16:28:15,608 INFO [main] hbase.TestGlobalMemStoreSize(125): TestGlobalMemStoreSize,m\xED\xED\xED\xE7,1322094494563.6639f54133a3ef8c6b11a9491030f7b6. 0
2011-11-23 16:28:15,608 INFO [main] hbase.TestGlobalMemStoreSize(125): TestGlobalMemStoreSize,zzzzz,1322094494579.48c6f4e6614fc095fdfe44469df28b5a. 0
{code}
Looks like the following should be performed outside the for loop:
{code}
// If meta, flush it last
if (meta != null) flush(meta, server);
{code}
This means the assertion should be done in a separate for loop.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156653#comment-13156653 ]
Hudson commented on HBASE-4853:
-------------------------------
Integrated in HBase-TRUNK #2477 (See [https://builds.apache.org/job/HBase-TRUNK/2477/])
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
HBASE-4853 HBASE-4789 does overzealous pruning of seqids; REVERT TEMPORARILY TO GET TED COMMENT IN
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156395#comment-13156395 ]
stack commented on HBASE-4853:
------------------------------
I think its something other than timeout and my patch (or my patch uncovers messed-up ness that was going on here). Let me keep at it.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-trunk.txt
Trunk is different here.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156368#comment-13156368 ]
Ted Yu commented on HBASE-4853:
-------------------------------
By increasing timeout to 6 seconds (Pardon me, N), I wasn't able to reproduce failure in TestGlobalMemStoreSize after 20 iterations:
{code}
Index: src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
===================================================================
--- src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java (revision 1205638)
+++ src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java (working copy)
@@ -100,11 +100,12 @@
}
LOG.info("Post flush on " + server.getServerName());
long now = System.currentTimeMillis();
- long timeout = now + 3000;
+ long timeout = now + 6000;
while(server.getRegionServerAccounting().getGlobalMemstoreSize() != 0 &&
timeout < System.currentTimeMillis()) {
Threads.sleep(10);
}
+ LOG.info("About to check GlobalMemstoreSize");
assertEquals("Server=" + server.getServerName() + ", i=" + i++, 0,
server.getRegionServerAccounting().getGlobalMemstoreSize());
}
{code}
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156356#comment-13156356 ]
Ted Yu commented on HBASE-4853:
-------------------------------
With patch v5, I got the following:
{code}
testGlobalMemStore(org.apache.hadoop.hbase.TestGlobalMemStoreSize) Time elapsed: 11.516 sec <<< FAILURE!
java.lang.AssertionError: Server=10.246.204.31,62993,1322086547613, i=0 expected:<0> but was:<608>
{code}
Here is tail of test output:
{code}
2011-11-23 14:15:55,955 INFO [main] regionserver.Store(631): Added hdfs://localhost:62971/user/zhihyu/.META./1028785192/info/6d51d01d9498464eb025ca045e696ce4, entries=47, sequenceid=36, filesize=8.4k
2011-11-23 14:15:55,956 INFO [main] regionserver.HRegion(1396): Finished memstore flush of ~17.2k/17608 for region .META.,,1.1028785192 in 44ms, sequenceid=36, compaction requested=false
2011-11-23 14:15:55,956 INFO [main] hbase.TestGlobalMemStoreSize(99): Flush .META.,,1.1028785192 on 10.246.204.31,62993,1322086547613, false, size=608
2011-11-23 14:15:55,957 INFO [main] hbase.TestGlobalMemStoreSize(99): Flush TestGlobalMemStoreSize,,1322086555196.e2b7276e785c7f6213a5bdd08a54cf8e. on 10.246.204.31,62993,1322086547613, false, size=608
2011-11-23 14:15:55,957 INFO [main] hbase.TestGlobalMemStoreSize(99): Flush TestGlobalMemStoreSize,c,P\xE3+,1322086555201.2c847584e6af6e64f3bae631bd722934. on 10.246.204.31,62993,1322086547613, false, size=608
2011-11-23 14:15:55,957 INFO [main] hbase.TestGlobalMemStoreSize(99): Flush TestGlobalMemStoreSize,q\x83\xCC\xF1{,1322086555217.f5079469f9fa696de61b9db6364cd6e7. on 10.246.204.31,62993,1322086547613, false, size=608
2011-11-23 14:15:55,957 INFO [main] hbase.TestGlobalMemStoreSize(101): Post flush on 10.246.204.31,62993,1322086547613
{code}
Basically there was no mentioning of flush completion for TestGlobalMemStoreSize table.
I think we should add a log before the assertion so that we know how long we spent waiting in the while loop:
{code}
assertEquals("Server=" + server.getServerName() + ", i=" + i++, 0,
server.getRegionServerAccounting().getGlobalMemstoreSize());
{code}
We should increase the wait time beyond 3 seconds.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v9.txt
Thanks for the testing Ted. Yeah, I noticed that after uploading patch myself. Here is another patch that waits even less time than the 3 seconds that used to be in place but if we find that not all memstores are flushed, we assert the problem region is meta, tna then we give the meta another flush on way out to see if that'll clear mem. Adds better debugging output too. Works for me ten times in a row.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Assignee: stack
Status: Open (was: Patch Available)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155598#comment-13155598 ]
stack commented on HBASE-4853:
------------------------------
Here is the manifestation of our failing to clean up WALs:
{code}
2011-11-23 00:08:31,155 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=36, maxlogs=32; forcing flush of 1 regions(s): 9e22583862f34489f04c42e1f3d16bce
2011-11-23 00:08:31,155 WARN org.apache.hadoop.hbase.regionserver.LogRoller: Failed to schedule flush of 9e22583862f34489f04c42e1f3d16bce, region=null, requester=null
{code}
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Open (was: Patch Available)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156381#comment-13156381 ]
stack commented on HBASE-4853:
------------------------------
Nvm. I ran it a few times. There is something going on here. Says N bytes but we are not flushing. Let me take a look.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156451#comment-13156451 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504941/4853-v6.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -162 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/353//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/353//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/353//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v5.txt
Here's a fix. I need a review given how this patch is actually revert of two commits I've made -- one recent and another a couple of months ago.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155660#comment-13155660 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504841/4853--no-prefix.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/342//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-4853:
--------------------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
0.92.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Closing as it was committed.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v4.txt
Working patch. Not done yet. Also has unit test to show hole (an edit is getting in and its seqid is sticking around in lastSeqid for the region w/o being cleared).
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155718#comment-13155718 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504846/4853-trunk.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -162 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/344//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/344//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/344//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853.txt
Here is patch that changes the blanket purge of our region from lastseqwritten map on flush completion to only remove it when no edits have come in in meantime (as it used to be in 0.90.x).
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v10.txt
What I applied (had to do little wiggling to get it into 0.92). Applied branch and 0.92.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155578#comment-13155578 ]
Jean-Daniel Cryans commented on HBASE-4853:
-------------------------------------------
An example of data loss.
First, we see a flush with seqid being 1389 but the message "Why is there a raw encodedRegionName in lastSeqWritten" means it deleted 1389.
{quote}
2011-11-22 11:21:08,526 WARN [RegionServer:1;hbasedev,54530,1321989619241.cacheFlusher] wal.HLog(1364): Why is there a raw encodedRegionName in lastSeqWritten? name=a9b1d96a4554f545a74142bc42f8c48e, seqid=1389
2011-11-22 11:21:08,526 INFO [RegionServer:1;hbasedev,54530,1321989619241.cacheFlusher] regionserver.HRegion(1259): Finished memstore flush of ~654.3k for region test,,1321989625594.a9b1d96a4554f545a74142bc42f8c48e. in 493ms, sequenceid=1388, compaction requested=false
{quote}
HLog roll happens, it sees 0 last seq id so it clears all the WALs.
{quote}
2011-11-22 11:21:08,932 INFO [RegionServer:1;hbasedev,54530,1321989619241.logRoller] wal.HLog(582): Roll /user/jdcryans/.logs/hbasedev,54530,1321989619241/hbasedev%2C54530%2C1321989619241.1321989667490, entries=41, filesize=25596. for /user/jdcryans/.logs/hbasedev,54530,1321989619241/hbasedev%2C54530%2C1321989619241.1321989668526
2011-11-22 11:21:08,932 DEBUG [RegionServer:1;hbasedev,54530,1321989619241.logRoller] wal.HLog(593): Last sequenceid written is empty. Deleting all old hlogs
{quote}
The last one had max seq id as 1424, which was never flushed.
{quote}
2011-11-22 11:21:08,967 INFO [RegionServer:1;hbasedev,54530,1321989619241.logRoller] wal.HLog(823): moving old hlog file /user/jdcryans/.logs/hbasedev,54530,1321989619241/hbasedev%2C54530%2C1321989619241.1321989667490 whose highest sequenceid is 1424 to /user/jdcryans/.oldlogs/hbasedev%2C54530%2C1321989619241.1321989667490
{quote}
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4853:
--------------------------
Comment: was deleted
(was: By increasing timeout to 6 seconds (Pardon me, N), I wasn't able to reproduce failure in TestGlobalMemStoreSize after 20 iterations:
{code}
Index: src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
===================================================================
--- src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java (revision 1205638)
+++ src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java (working copy)
@@ -100,11 +100,12 @@
}
LOG.info("Post flush on " + server.getServerName());
long now = System.currentTimeMillis();
- long timeout = now + 3000;
+ long timeout = now + 6000;
while(server.getRegionServerAccounting().getGlobalMemstoreSize() != 0 &&
timeout < System.currentTimeMillis()) {
Threads.sleep(10);
}
+ LOG.info("About to check GlobalMemstoreSize");
assertEquals("Server=" + server.getServerName() + ", i=" + i++, 0,
server.getRegionServerAccounting().getGlobalMemstoreSize());
}
{code})
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Open (was: Patch Available)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159788#comment-13159788 ]
Hudson commented on HBASE-4853:
-------------------------------
Integrated in HBase-0.92 #163 (See [https://builds.apache.org/job/HBase-0.92/163/])
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155771#comment-13155771 ]
Hadoop QA commented on HBASE-4853:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504860/4853-v4.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/346//console
This message is automatically generated.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156631#comment-13156631 ]
Hudson commented on HBASE-4853:
-------------------------------
Integrated in HBase-TRUNK-security #7 (See [https://builds.apache.org/job/HBase-TRUNK-security/7/])
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
HBASE-4853 HBASE-4789 does overzealous pruning of seqids; REVERT TEMPORARILY TO GET TED COMMENT IN
HBASE-4853 HBASE-4789 does overzealous pruning of seqids
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v10.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155611#comment-13155611 ]
Jean-Daniel Cryans commented on HBASE-4853:
-------------------------------------------
TestReplication was :) but it's not its main purpose. It might be doing too much.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853-v6.txt
Same patch with better variable naming.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Patch Available (was: Open)
Trigger build on last version.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853-v8.txt, 4853-v9.txt, 4853-v9.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Attachment: 4853--no-prefix.txt
no prefix version
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Open (was: Patch Available)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4853:
-------------------------
Status: Open (was: Patch Available)
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853-v7.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4853) HBASE-4789 does overzealous pruning
of seqids
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156370#comment-13156370 ]
Ted Yu commented on HBASE-4853:
-------------------------------
We should let TestGlobalMemStoreSize pass consistently. HBASE-4722 tried to solve this issue.
> HBASE-4789 does overzealous pruning of seqids
> ---------------------------------------------
>
> Key: HBASE-4853
> URL: https://issues.apache.org/jira/browse/HBASE-4853
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 4853--no-prefix.txt, 4853-trunk.txt, 4853-v4.txt, 4853-v5.txt, 4853-v6.txt, 4853.txt
>
>
> Working w/ J-D on failing replication test turned up hole in seqids made by the patch over in hbase-4789. With this patch in place we see lots of instances of the suspicious: 'Last sequenceid written is empty. Deleting all old hlogs'
> At a minimum, these lines need removing:
> {code}
> diff --git a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> index 623edbe..a0bbe01 100644
> --- a/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> +++ b/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
> @@ -1359,11 +1359,6 @@ public class HLog implements Syncable {
> // Cleaning up of lastSeqWritten is in the finally clause because we
> // don't want to confuse getOldestOutstandingSeqNum()
> this.lastSeqWritten.remove(getSnapshotName(encodedRegionName));
> - Long l = this.lastSeqWritten.remove(encodedRegionName);
> - if (l != null) {
> - LOG.warn("Why is there a raw encodedRegionName in lastSeqWritten? name=" +
> - Bytes.toString(encodedRegionName) + ", seqid=" + l);
> - }
> this.cacheFlushLock.unlock();
> }
> }
> {code}
> ... but above is no good w/o figuring why WALs are not being rotated off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira