You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gopinathan A (Created) (JIRA)" <ji...@apache.org> on 2012/04/13 16:32:16 UTC
[jira] [Created] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Not all the regions are getting assigned after the log splitting.
-----------------------------------------------------------------
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Priority: Critical
Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5782:
--------------------------------------
Summary: Edits can be appended out of seqid order since HBASE-4487 (was: Not all the regions are getting assigned after the log splitting.)
Changing the summary to something more relevant.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255828#comment-13255828 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Anybody opposed to do my patch for 0.94 and do a rewrite in trunk? Todd? Stack? Ted?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254646#comment-13254646 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
Found the problem and working on soln. Infact this problem would have introduced data loss easily in simple RS going down scenario.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255034#comment-13255034 ]
stack commented on HBASE-5782:
------------------------------
I just committed a tool over on HBASE-5792. It tests WALs. If you pass the -verify flag, you'll see that even w/ just three threads, sequence ids are out of order. Could be useful verifying whatever fix we have here.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256087#comment-13256087 ]
stack commented on HBASE-5782:
------------------------------
@Lars As to your patch being 'slower' when fewer threads, I think you can't do such a compare. W/o your patch, we are broke.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255937#comment-13255937 ]
stack commented on HBASE-5782:
------------------------------
bq. We won't write more into the log (once we take the pendingWrites they are gone
Is that so? We don't get the pendingWrites until we are under the flush lock but we've taken doneUpTo before we go under the lock.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256978#comment-13256978 ]
Hudson commented on HBASE-5782:
-------------------------------
Integrated in HBase-0.94-security #15 (See [https://builds.apache.org/job/HBase-0.94-security/15/])
HBASE-5782 Edits can be appended out of seqid order since HBASE-4487 (Revision 1327672)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255192#comment-13255192 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522884/5782-sketch.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause mvn compile goal to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1547//testReport/
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1547//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254433#comment-13254433 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Thanks Ram!
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255770#comment-13255770 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Yep. It's crucial that syncTillHere is updated last.
The pendingWrites are appended strictly in order, so there is a very short race that might lead to sync be issued multiple time when only one was necessary (it seems the same race condition existed before).
So I think it is safe and low risk.
The question now is: Do this for 0.94 and then a more elaborate rewrite in trunk, or do a more risky rewrite for 0.94?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255832#comment-13255832 ]
Todd Lipcon commented on HBASE-5782:
------------------------------------
Seems reasonable to me. Did yours pass tests, etc?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254925#comment-13254925 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
So the problem is that logSyncerThread keeps the edit in order but the syncer then applies the pending batches potentially out of order?
We might just need a "sync lock" to prevent two threads syncing at the same time. This is different from the update lock, which also prevents writing any edits.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255168#comment-13255168 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Is this really only a problem because of HBASE-4528 and HBASE-4487?
It seems even without HLog.appendNoSync() is is possible that one threads flushes an entire batch of pending write before an thread that started earlier can get to it.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253418#comment-13253418 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
@Lars
I was checking this issue. I think it has to be fixed before RC? What you say?
Not sure of the root cause still.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255239#comment-13255239 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522890/5782.unfinished-stack.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1548//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1548//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1548//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254959#comment-13254959 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
bq. I'd prefer solution that doesn't add a lock to patch something that's broken.
I agree.
I suggest the following actions:
1. pull back HBASE-4487 in 0.94 and trunk
2. a) spend major effort on HBASE-5699 (multiple WALs per region server)
2. b) make HBASE-4487 semantically correct
2.a and 2.b can proceed in parallel. But I think HBASE-5699 is the long-term solution.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255974#comment-13255974 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
@Stack: I ran it too. It works fine. The "interesting" thing I find that it is faster *with* the patch! And that scares me.
I ran with 100 threads, 10000 iterations. With the patch it took 29s, without it took 43s. This is on a 6 core machine with hyper threading.
Now, the HLogPerformanceEvaluation does not work with -nosync and -verify, right? (presumably because no final sync is issued).
Wouldn't mind to get some other numbers from you as well if you had some time.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-5782:
------------------------------------------
Status: Patch Available (was: Open)
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256043#comment-13256043 ]
stack commented on HBASE-5782:
------------------------------
I made the hlog perf tool work on hdfs and ran some basic tests. Both Todd an Lars' patches seem faster than what we have currently.
Running w/o a fix on hdfs w/ current trunk I have to disable verify because it fails (verify happens after we print out test timings).
$ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -conf /home/stack/hadoop-conf/core-site.xml -path hdfs://sv4r11s38:7000/tmp -threads 100 -roll 10000
12/04/17 22:58:28 INFO wal.HLogPerformanceEvaluation: Summary: threads=100, iterations=10000 took 100.630s 9937.395ops/s
12/04/17 23:00:33 INFO wal.HLogPerformanceEvaluation: Summary: threads=100, iterations=10000 took 94.945s 10532.413ops/s
Todd patch on hdfs:
$ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -conf /home/stack/hadoop-conf/core-site.xml -path hdfs://sv4r11s38:7000/tmp -threads 100 -roll 10000 -verify
12/04/17 22:53:35 INFO wal.HLogPerformanceEvaluation: Summary: threads=100, iterations=10000 took 81.202s 12314.967ops/s
Lars patch:
12/04/17 23:07:08 INFO wal.HLogPerformanceEvaluation: Summary: threads=100, iterations=10000 took 76.800s 13020.833ops/s
For Todd and Lars, both pass verify which checks that seqids are ordered and that we wrote as much as we think we did.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-5782:
-------------------------------
Attachment: 5782-sketch.txt
half-done patch of a proposed solution... got to run at the moment but will come back later with something more complete. The idea is to only have one "seqnum" in use, and make sure only a single thread pushes from the pending edits queue into a seqfile writer at a time.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254782#comment-13254782 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
This patch passes TestHLog and TestHRegion. Not able to run the testsuite as at home.
The idea here is the edits till the seq id already persisted can be safely ignored.
Now from this issue we are sure that the edits can appear in diff seq not necessary in the ascending order.
So all the edits less than the seq id from the store files are skipped and all other edits are consdiered. Please provide your comments.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256058#comment-13256058 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Thanks Stack!
I'll run a test with fewer threads too, just to make sure, the fact that both of our patches are faster probably means that we did a lot of unnecessary sync'ing before...? 20% performance increase is pretty damn awesome.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5782:
---------------------------------
Attachment: 5782-lars-v2.txt
In the interest of keeping this simple, here's another simple patch.
This patch assumes that:
# It is only the appends that need to happen in order.
# The sync is the time consuming operation.
So a special lock is only held for enforce the ordering of the appends (appends are single threaded) and syncs can happen in parallel without this lock held.
This should give us the maximum concurrency possible, while keeping the change small and palletable for 0.94.
Please let me know whether I am smoking something.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255102#comment-13255102 ]
stack commented on HBASE-5782:
------------------------------
The test passes w/ your patch in place Lars. Let me try a perf compare.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254669#comment-13254669 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
We can easily reproduce this problem. I think it applies to any edit that happens and if it is synced by the log syncer thread.
So we must have some sync mechanism here to ensure this problem does not happen.
Based on one of our requirements i was thinking that the log syncer thread should be configurable like either to use it or not use it.
Please provide your suggestions.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254805#comment-13254805 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
bq. Based on one of our requirements i was thinking that the log syncer thread should be configurable like either to use it or not use it.
As long as it's enabled by default I'm good with that.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257005#comment-13257005 ]
Hudson commented on HBASE-5782:
-------------------------------
Integrated in HBase-0.94 #129 (See [https://builds.apache.org/job/HBase-0.94/129/])
HBASE-5782 Edits can be appended out of seqid order since HBASE-4487 (Revision 1327672)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255344#comment-13255344 ]
stack commented on HBASE-5782:
------------------------------
@Ram Read over the HLog comments. Its got stuff on why we want sequenceids in order and where we have dependency on their being ordered, at least they are notes on how we used to think. I was wondering too about ordering today. If we didn't have to have order, then it would make stuff like running a regionserver with N WALs a bit easier, and we don't try to guarantee sequence order when replicating. But I'm wary undoing order though without our giving the issue a bunch of thought first (Your patch above makes me nervous).
On the patch, Todds' seems way superior to me. His is more radical, removing what seems to be a confusing sequenceid double, and its more clear whats going on.
Oh, and thanks to you fellas for finding this one. Its a good one.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254892#comment-13254892 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
Looking more into this, I think HBASE-4487 is the real issue. I think I can also prove that you can get the issue even with a disabled {{LogSyncer}}.
t1 does {{appendNoSync}} of k1
t1 does {{syncer}} up to {{getPendingWrites}}
t2 does {{appendNoSync}} of k2
t2 does {{syncer}} up to the end
In the log you'd see k2 then k1 so what's really wrong to me is this:
{code}
// Done in parallel for all writer threads, thanks to HDFS-895
List<Entry> pending = logSyncerThread.getPendingWrites();
{code}
Although accessing pending writes is done in sync, you can apply them in whichever way.
Furthermore, {{logSyncerThread.hlogFlush}} can also append entries to the WAL in any order. For example, if both t1 and t2 have multiple edits they could end up intermingled in the WAL simply by doing {{hlogFlush}} at the same time.
If {{LogSyncer}} was really an issue then {{HRegion.put}} and {{HRegion.delete}} would need to be disabled too since they don't use {{appendNoSync}} and just sync everything :)
How this used to work is that threads could only append to the WAL under the {{updateLock}} and that was done at the same time as the {{doWrite}} which creates the key. The call to sync could be done by any number of threads at the same time.
If this is right, then we should pull back HBASE-4487 or add more locks.
We should also change this Jira's title once we get a better understanding of the problem because it's not a region assignment problem.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255172#comment-13255172 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
bq. It seems even without HLog.appendNoSync() is is possible that one threads flushes an entire batch of pending write before an thread that started earlier can get to it.
We didn't have pending writes before, it was inside the sequence file writer, and we append under lock. Managing those pending writes is what's giving us trouble.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253996#comment-13253996 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
And is there anything interesting in the log files?
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255760#comment-13255760 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522975/5782-lars-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1551//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1551//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1551//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255329#comment-13255329 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
@Lars
Correct me if am wrong.
Currently the edits are ordered atleast in the memory so that the store file that is persisted is having the proper sequence number.
But it is when applying it to the WAL there is a change and the same needs to be taken care while replaying.
Is there any special consideration needed that my edits have to ordered even in the WAL file? What is the design consideration behind this?
May be am missing something here.
@Stack
This patch is one similar to the one i had in mind. Synchronize getPendingWrites and the 'writer.append(e);'.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255299#comment-13255299 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
Thanks to all for working on this.
@JD
I agree that LogSyncer thread is not the real problem, it can happen even without that.
My initial thought was also to introduce some locks but finally thought that the expected parallelism cannot be obtained. That is why i went with the fix while replayingtheRecoveredEdits. I will go thro the current patch also.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256220#comment-13256220 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523074/5782-v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1560//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1560//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1560//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5782:
---------------------------------
Fix Version/s: 0.94.0
Assigning to 0.94.0 for now
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5782:
------------------------------
Comment: was deleted
(was: {code}
+ synchronized (flushLock) {
+ List<Entry> pending;
- // write out all accumulated Entries to hdfs.
- for (Entry e : pending) {
- writer.append(e);
+ synchronized (this) {
{code}
Is the second synchronized needed ? )
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257105#comment-13257105 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Yes.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5782:
---------------------------------
Attachment: 5782-v3.txt
This is would I would commit to 0.94. (plus a test that either Stack or I would write).
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257100#comment-13257100 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
Do we need to log another issue for trunk which would finish Todd's work ?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256185#comment-13256185 ]
stack commented on HBASE-5782:
------------------------------
Want me to make a test that does simple three threads with just a few edits ... say 1k... and then verifies all in order and all edits written so we notice regression?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255718#comment-13255718 ]
Todd Lipcon commented on HBASE-5782:
------------------------------------
Don't think you're smoking something. It looks reasonable, though I think there are still races around the update of doneUpTo, syncedTilHere, etc. These races may be "safe" in that they only result in too-low values (and thus extra needless syncs) but I'm not 100% sure of it. Maybe we can extend the verification test case to run as a stress test, including rolls, and run it overnight or something?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5782) Edits can be appended
out of seqid order since HBASE-4487
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256203#comment-13256203 ]
Lars Hofhansl edited comment on HBASE-5782 at 4/18/12 5:03 AM:
---------------------------------------------------------------
This is what I would commit to 0.94. (plus a test that either Stack or I would write).
was (Author: lhofhansl):
This is would I would commit to 0.94. (plus a test that either Stack or I would write).
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255395#comment-13255395 ]
Todd Lipcon commented on HBASE-5782:
------------------------------------
Looks like some test timeouts... will investigate tomorrow.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5782:
-------------------------
Attachment: 5782.unittest.txt
A unit test. Currently fails.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255834#comment-13255834 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
HLogPerformanceEvaluation verification is needed as well.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254994#comment-13254994 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
HLog.appendNoSync() is used by HRegion.{append|doMiniBatchPut|mutateRowsWithLocks}.
Those methods would be affected when HLog.appendNoSync() is removed.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256072#comment-13256072 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Did some more tests with HLogPerformanceEvaluation (and my patch):
10 threads, 100000 iterations: with patch: 42s, without patch: 41s
5 threads, 200000 iterations: with patch: 46s, without patch: 44s
2 threads, 200000 iterations: with patch: 46s, without patch: 44s
So for fewer threads my patch is slightly slower.
So... What do we do? :)
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255877#comment-13255877 ]
stack commented on HBASE-5782:
------------------------------
Looking at Lars patch.
On you 1. and 2. above, apparently the append is also expensive according to Dhruba. Just saying.
Also on.... "...might lead to sync be issued multiple time when only one was necessary (it seems the same race condition existed before)."
Yes, this we have always had.
I'd say kill this stuff... it looks like rubbish to me:
{code}
+ syncBatchSize.addAndGet(doneUpto - this.syncedTillHere);
{code}
Its not read by anyone, looks like the math can go wonky, and when it is read, its set back to zero which is probably unexpected. Kill it I'd say.
I think this is ok:
{code}
+ this.syncedTillHere = Math.max(this.syncedTillHere, doneUpto);
{code}
but this is racy
{code}
long doneUpto = this.unflushedEntries.get();
{code}
It could be low in number; i.e. we could be putting into hdfs more edits than the current value of unflushedEntries if we read after an edit has been added to the queue but before the above is updated. Is that ok? Its ok if this is a little sloppy especially if it under reports?
On tactic for 0.94, sure on doing this for 0.94 though I like Todds fix better. The verification tool will help you figure if this slows stuff much and if we are writing out of order. Let me know if you want me to run it for you. Let me add in log rolling too as per Todd suggestion.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257263#comment-13257263 ]
Hudson commented on HBASE-5782:
-------------------------------
Integrated in HBase-TRUNK-security #175 (See [https://builds.apache.org/job/HBase-TRUNK-security/175/])
HBASE-5782 Edits can be appended out of seqid order since HBASE-4487 (Revision 1327673)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256993#comment-13256993 ]
Hudson commented on HBASE-5782:
-------------------------------
Integrated in HBase-TRUNK #2783 (See [https://builds.apache.org/job/HBase-TRUNK/2783/])
HBASE-5782 Edits can be appended out of seqid order since HBASE-4487 (Revision 1327673)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-5782:
-------------------------------
Attachment: hbase-5782.txt
New rev - this one should even compile! :)
I tested a few of the hlog tests locally and they seem to pass. Also ran the standalone hlog tool with a bunch of threads and -verify. Throughput is identical, and errors are gone.
I'm nervous about error handling, though... seems like if the syncer thread had an error, another thread might wait forever.
I'd really like to rip this apart and make it all component-ized and unit-testable, but tried to keep this patch minimal.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256030#comment-13256030 ]
Todd Lipcon commented on HBASE-5782:
------------------------------------
I think for 0.94.0 we should commit Lars's, and file another JIRA to clean up / rewrite this stuff to be less circuitous. Could definitely do with a cleanup.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256933#comment-13256933 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Now note, that TestHLog still does not run anything locally (neither in 0.94 or trunk), there is something with that specific test it seems.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254728#comment-13254728 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
@Zhihong
Yes HBASE-5699 can provide high write throughput. But even there the syncing should be done for every WAL that we will be writing there we should ensure that we do not sync everytime holding some locks.
So ideally this problem is coming out of parallel syncs(which may happen even if we have > 1 WAL). Correct me if am wrong.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256200#comment-13256200 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Yes, we need a test for this. What scared me most about this bug was that no test caught it, and this one of *the* core parts of HBase.
This test would basically just call code from HLogPerformanceEvaluation, right?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255130#comment-13255130 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
bq. Don't hate me, just throwing this out there.
LOL
My current concerns:
- Performance, and I'm not the only one concerned.
- OOME, what happens if you enable deferred log flush and HDFS is slowing down? That's actually an issue with HBASE-4487 since the {{LinkedList}} of pending writes is unbounded whereas before simply appending to the file would slow you down.
- This patch will make it that threads can sync data from threads that came in later. You need to check {{txid <= this.syncedTillHere}} again once you are past the {{synchronized(syncLock)}} and return if it was taken care of while you were blocking.
- Deadlocks, you never know when adding new locks :)
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254385#comment-13254385 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
@LArs
Tomorrow i can try writing a test with similar scenario.
bq.(Although that might be difficult)
I think it should be easier. Hopefully tomorrow can come up with some initial analysis.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255094#comment-13255094 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
The short term choices we have are:
# revert HBASE-4528, HBASE-4487, and HBASE-5541 (are there others?)
# Partially revert HBASE-2467 (or devise other ways to have strictly one thread flushing an HLog).
Maybe Todd as the author of HBASE-2467 could chime in... Todd?
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255191#comment-13255191 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Sketched patch looks good, making sure that we do sync' a batch of write before the previous batch is done.
I was thinking about something similar, but still allowing multiple threads to write, just that a thread cannot start writing before the previous batch is confirmed sync'ed. I guess we actually wouldn't get more parallelism out that than with your approach.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253990#comment-13253990 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
You tried multiple times in 0.92.1, and no problem?
Any chance to capture this in a test? (Although that might be difficult)
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5782:
---------------------------------
Attachment: 5782.txt
Simple patch to ensure only one thread flushes the log.
Don't hate me, just throwing this out there.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256968#comment-13256968 ]
stack commented on HBASE-5782:
------------------------------
Sorry. Dumb. The tool calls system.exit. Let me fix in another issue.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5782:
---------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.94 and 0.96 (including test)
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5782) Edits can be appended
out of seqid order since HBASE-4487
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256094#comment-13256094 ]
Lars Hofhansl edited comment on HBASE-5782 at 4/18/12 12:15 AM:
----------------------------------------------------------------
I'm +1 on committing on my patch for 0.94.0. We can then either revisit for 0.94.1 or 0.96.
was (Author: lhofhansl):
I'm +1 on committing on my patch for 0.94.0. We can then either revisit for 0.94.1+ or 0.96.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255939#comment-13255939 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Ah, I see. Yes, doneUpto line should be pulled into the synchronized block.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256061#comment-13256061 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
For my part I tested on a cluster with 1kb values, huge batches, and I don't see any improvement with Lars' patch. Todd's peaks 20% higher.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255959#comment-13255959 ]
stack commented on HBASE-5782:
------------------------------
@Lars But it own't matter right since the map we are getting from is not under our new flush lock? I think its harmless. We will undercount whats been flushed I believe; we'll not overcount (and so possible lose data)?
I added log rolling and tested your patch using HLogPerformanceEvaluation. It 'works' at least. If you want me to compare before and after, just say.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255135#comment-13255135 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522867/5782.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1545//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1545//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1545//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255885#comment-13255885 ]
stack commented on HBASE-5782:
------------------------------
Can we try and make Todd's work? It does some nice cleanup.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256202#comment-13256202 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
I can work on a test, unless you like to Stack, or maybe you have started on it anyway.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-5782:
------------------------------------------
Priority: Blocker (was: Critical)
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255308#comment-13255308 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
@Ram: It looks like your patch should work for this scenario, but I'd be generally wary about edits not being in order in the WAL.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256044#comment-13256044 ]
stack commented on HBASE-5782:
------------------------------
Ok on lars patch into 0.94.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256788#comment-13256788 ]
stack commented on HBASE-5782:
------------------------------
You will need to pull in HLogPerformanceEvaluation. Copy it whole (don't do the hbase-5792 because it got mod'd a few times subsequent to commit). You could also just commit the unit test to trunk and not to 0.94; that should be fine long as we hold to committing patches to trunk first.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-5782:
------------------------------------
Assignee: Lars Hofhansl (was: ramkrishna.s.vasudevan)
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253886#comment-13253886 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
Did you have HBASE-5778 enabled?
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254666#comment-13254666 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
The problem here is
{code}
HLogKey logKey = makeKey(encodedRegionName, tableName, seqNum, now, clusterId);
doWrite(info, logKey, edits, htd);
{code}
We form a key here which is unique based on the seqNum. Now as we do appendNoSync() there is no sync called in append flow. But there is always the LogSyncer thread which periodically syncs.
All the edits are added to
{code}
synchronized void append(Entry e) throws IOException {
pendingWrites.add(e);
}
{code}
Here the pendingWrites are obtained and then it is flushed one by one.
{code}
pending = logSyncerThread.getPendingWrites();
// First flush all the pending writes to HDFS. Then
// issue the sync to HDFS. If sync is successful, then update
// syncedTillHere to indicate that transactions till this
// number has been successfully synced.
logSyncerThread.hlogFlush(tempWriter, pending);
{code}
Here there is no synchronization and hence the pendingwrites are not in sequence. Because it is not in sequence the recovered.edits file is also created with no sequence in some cases.
{code}
if (key.getLogSeqNum() <= currentEditSeqId) {
LOG.debug("******** key is " + key.getLogSeqNum()
+ "current edit is " + currentEditSeqId);
skippedEdits++;
continue;
}
{code}
I just added the above log while replayRecoveredEdits and i can find
{code}
******** key is 175current edit is 176
******** key is 174current edit is 176
******** key is 186current edit is 187
******** key is 188current edit is 190
******* key is 189current edit is 190
******** key is 204current edit is 205
******** key is 206current edit is 207
******** key is 217current edit is 218
******** key is 216current edit is 218
******** key is 255current edit is 256
{code}
This leads to dataloss. Correct me if am wrong.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254810#comment-13254810 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522794/HBASE-5782.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//console
This message is automatically generated.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256756#comment-13256756 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
For some reason I cannot run TestHLog locally. I always get:
{code}
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
{code}
I can run all other tests that I tried (including some others marked with @LargeTests). Verified again with HLogPerformanceEvaluation manually.
Should we put HBASE-5792 in 0.94 as well, so that we use this test?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255121#comment-13255121 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
Interesting patch.
How do we measure / compare the following combinations:
1. HLog.appendNoSync() used with one sync thread doing flush
2. HLog.appendNoSync() not used, multiple sync threads doing flush in parallel
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5782:
-------------------------
Attachment: 5782.unfinished-stack.txt
I started in on a patch but I prefer Todd's. Mine just put together the getting of pending writes and their appending under same sync block but left sync outside of the synchronization. Unfinished.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254917#comment-13254917 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
bq. If this is right, then we should pull back HBASE-4487 or add more locks
Adding more locks would take more time to validate / test.
In order to get 0.94.0 out the door, can we pull back HBASE-4487 in 0.94 and pursue the locking approach in trunk (or separate branch) ?
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255211#comment-13255211 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
{code}
+ synchronized (flushLock) {
+ List<Entry> pending;
- // write out all accumulated Entries to hdfs.
- for (Entry e : pending) {
- writer.append(e);
+ synchronized (this) {
{code}
Is the second synchronized needed ?
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan reassigned HBASE-5782:
---------------------------------------------
Assignee: ramkrishna.s.vasudevan
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Gopinathan A (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253978#comment-13253978 ]
Gopinathan A commented on HBASE-5782:
-------------------------------------
Initially I also doubted the same. I am able to reproduce same issue after disabling the wal compression.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Priority: Critical
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256187#comment-13256187 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
+1 on such a test which would prevent regression.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255784#comment-13255784 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
bq. The pendingWrites are appended strictly in order, so there is a very short race that might lead to sync be issued multiple time when only one was necessary (it seems the same race condition existed before).
A short race is better than what we currently do in 0.92 (and before) where everything syncs everything.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-5782:
------------------------------------------
Attachment: HBASE-5782.patch
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255381#comment-13255381 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522913/hbase-5782.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1549//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1549//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1549//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256176#comment-13256176 ]
ramkrishna.s.vasudevan commented on HBASE-5782:
-----------------------------------------------
I'm +1 on Lar's patch.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255911#comment-13255911 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
@Stack: the doneUpTo is fine, because it is only used to set syncedTillHere. We won't write more into the log (once we take the pendingWrites they are gone), but we might sync too much unnecessarily.
Will do the (performance) testing now. I don't have a cluster at my disposal atm, so I'll do it with a local HDFS instance.
If we'd rather finish Todd's for 0.94 that'd be nice.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256085#comment-13256085 ]
stack commented on HBASE-5782:
------------------------------
I tried to reproduce what JD is seeing on cluster using same sized keys and values but Lars' patch completes before Todds. My test run may be too small I did thread dumps during Lars and Todd runs. Both seem to be down in sync mostly, down here 'org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3789)' otherwise hung up on sync points around wal append/sync.
Lets go w/ the Lars patch because minimal changes. As per Todd, lets file an issue to clean up this stuff with his patch as seed. From J-D work, any grease lightening we can apply around hlog append makes for a big difference in overall write throughput.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254932#comment-13254932 ]
Jean-Daniel Cryans commented on HBASE-5782:
-------------------------------------------
bq. In order to get 0.94.0 out the door, can we pull back HBASE-4487 in 0.94 and pursue the locking approach in trunk (or separate branch) ?
+1, we might want to review HBASE-4282 too as it seems to do something similar with the transaction ids.
bq. So the problem is that logSyncerThread keeps the edit in order but the syncer then applies the pending batches potentially out of order?
It's sad that the pending edits live in {{LogSyncer}}, that thread is really just suppose to call sync... but yeah so they are added there in order but then it's a free for all in {{syncer}}. Adding a sync there could solve the issue but in the end what it does is moving the log from appending (pre HBASE-4487) to syncing plus a _ton_ of new complexity in HLog. I'd prefer solution that doesn't add a lock to patch something that's broken.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5782:
------------------------------
Comment: was deleted
(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522884/5782-sketch.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause mvn compile goal to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1547//testReport/
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1547//console
This message is automatically generated.)
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256094#comment-13256094 ]
Lars Hofhansl commented on HBASE-5782:
--------------------------------------
I'm +1 on committing on my patch for 0.94.0. We can then either revisit for 0.94.1+ or 0.96.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256236#comment-13256236 ]
Hadoop QA commented on HBASE-5782:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523078/5782.unittest.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1562//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1562//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1562//console
This message is automatically generated.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: Lars Hofhansl
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 5782.txt, 5782.unfinished-stack.txt, 5782.unittest.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting
assigned after the log splitting.
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254724#comment-13254724 ]
Zhihong Yu commented on HBASE-5782:
-----------------------------------
One option is to not use appendNoSync() in doMiniBatchPut(). For high write throughput, I think HBASE-5699 should be the long-term solution.
> Not all the regions are getting assigned after the log splitting.
> -----------------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid
order since HBASE-4487
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255201#comment-13255201 ]
stack commented on HBASE-5782:
------------------------------
Not sure I follow but I do know this patch more ambitious than what I was at....
+ You remove the 'other' sequence numbering system, unflushedEntries? That looks good.
+ Are asserts on by default? We disabled them a while back I believe? You run w/ asserts? (Yeah, thats a good thing to test -- should you use your guava test instead?)
+ Its ugly we call it hlogFlush but internal we do appends (thats not your change)
+ I agree that the reset of the the pending writes linked list needs to be done under the synchronization held by hlogFlush
+ I like how you do pushback of edits if we fail....in hlogFlush.
+ On this thing:
{code}
+ // TODO: restore metric syncBatchSize.addAndGet(doneUpto - this.syncedTillHere);
{code}
Its not used anywhere and it the math looked dodgy... then when you read it it gets set to zero so I'm not so sure it is of any use.
> Edits can be appended out of seqid order since HBASE-4487
> ---------------------------------------------------------
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-sketch.txt, 5782.txt, HBASE-5782.patch
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira