You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (Created) (JIRA)" <ji...@apache.org> on 2011/11/21 21:56:51 UTC
[jira] [Created] (HBASE-4838) Port 2856 (TestAcidGuarantees is
failing) to 0.92
Port 2856 (TestAcidGuarantees is failing) to 0.92
-------------------------------------------------
Key: HBASE-4838
URL: https://issues.apache.org/jira/browse/HBASE-4838
Project: HBase
Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.92.0
Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159794#comment-13159794 ]
Hudson commented on HBASE-4838:
-------------------------------
Integrated in HBase-0.92 #163 (See [https://builds.apache.org/job/HBase-0.92/163/])
HBASE-4838 addendum, missing file
larsh :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157347#comment-13157347 ]
Ted Yu commented on HBASE-4838:
-------------------------------
Superb, Lars.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v2.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158086#comment-13158086 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Whoops, missing adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java.
Fixed in addendum patch.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158159#comment-13158159 ]
Hudson commented on HBASE-4838:
-------------------------------
Integrated in HBase-0.92-security #19 (See [https://builds.apache.org/job/HBase-0.92-security/19/])
HBASE-4838 addendum, missing file
larsh :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4838:
---------------------------------
Attachment: 4838-v3.txt
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155394#comment-13155394 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Still working on this. I have narrowed down the scenario (also happens with just 2 cells that are split), but don't have the root cause, yet.
I looked through the patch multiple times and I pretty sure the changes are correct (in the sense that they represent the patch), there must be an assumption about some behavior in trunk that is not present in 0.92.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4838) Port 2856
(TestAcidGuarantee is failing) to 0.92
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158086#comment-13158086 ]
Lars Hofhansl edited comment on HBASE-4838 at 11/28/11 12:52 AM:
-----------------------------------------------------------------
Whoops, missed adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java.
Fixed in addendum patch.
was (Author: lhofhansl):
Whoops, missing adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java.
Fixed in addendum patch.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158053#comment-13158053 ]
stack commented on HBASE-4838:
------------------------------
I'm good on it going in.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156350#comment-13156350 ]
Todd Lipcon commented on HBASE-4838:
------------------------------------
Maybe a problem with the HalfHFile references? After a compaction of the split daughters, does the doubling persist?
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4838:
---------------------------------
Attachment: 4838-v2.txt
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v2.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4838:
---------------------------------
Attachment: (was: 4838-v2.txt)
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156319#comment-13156319 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
With the above scenario what I found is this:
o the table is populated with only two KV: aaa and aab.
o after the split there two regions: ['', aaa) and [aaa,'')
x the client scanner first tries the 1st region
o then it tries the 2nd region
The X is where the difference is. In trunk (and unpatched 0.92), the region's internal scanner finds no KVs (as it should) and returns an empty result to the client scanner, which then proceeds to the next region.
In 0.92 with this patch, the region's internal scanner actually finds both aaa and aab in the 1st region (which is wrong), and then again the 2nd region (which is correct).
I don't know, yet, why this is happening, though. Maybe the scanner picks up the wrong store files, or there a problem with flushes or compactions.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157554#comment-13157554 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
@Stack: not sure. I do not have that run's output anymore. But when I run it again (multiple times) it passes (as long as I keep moving the mouse around - see HBASE-4874)
And this is in trunk already, only for 0.92.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157537#comment-13157537 ]
stack commented on HBASE-4838:
------------------------------
Wahoo!!
+1 on commit to trunk and 0.92
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-4838.
----------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to 0.92
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157538#comment-13157538 ]
stack commented on HBASE-4838:
------------------------------
The testClosing is same as I'm seeing on ubuntu?
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155596#comment-13155596 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
This still fails:
{noformat}
testFilterAcrossMultipleRegions(org.apache.hadoop.hbase.client.TestFromClientSid
e) Time elapsed: 12.233 sec <<< FAILURE!
java.lang.AssertionError: expected:<17576> but was:<28064>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.hadoop.hbase.client.TestFromClientSide.assertRowCount(Test
FromClientSide.java:528)
at org.apache.hadoop.hbase.client.TestFromClientSide.testFilterAcrossMul
tipleRegions(TestFromClientSide.java:436)
{noformat}
I went through the entire diff between 0.92 with this patch and trunk. There is nothing that sticks out that could be causing this.
This problem be easily reproduced by reducing the number of rows generated in HBaseTestingUtility.loadTable(...) to (note: replace 'z' with 'a' and 'b'):
{noformat}
public int loadTable(final HTable t, final byte[] f) throws IOException {
t.setAutoFlush(false);
byte[] k = new byte[3];
int rowCount = 0;
for (byte b1 = 'a'; b1 <= 'a'; b1++) {
for (byte b2 = 'a'; b2 <= 'a'; b2++) {
for (byte b3 = 'a'; b3 <= 'b'; b3++) {
k[0] = b1;
k[1] = b2;
k[2] = b3;
Put put = new Put(k);
put.add(f, null, k);
t.put(put);
rowCount++;
}
}
}
t.flushCommits();
return rowCount;
}
{noformat}
this will only generate two rows for this test, which makes it easier to debug.
The same test then fails with:
{noformat}
java.lang.AssertionError: expected:<2> but was:<4>
...
{noformat}
Somehow it is scanning more rows after a split that it is supposed to repeating some of the same rows. If somebody else wanted have a look at it, I could use another pair of eyes.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4838:
---------------------------------
Attachment: 4838-v1.txt
Patch identical to 0.92-patch in HBASE-2856.
This has issues with failing tests.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156369#comment-13156369 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Reference.java and HalfStoreFileReader.java are identical between 0.92 and trunk (and neither Reference nor HalfStoreFileReader appear in this patch), so that is likely not the cause.
I also verified now that it picks up the correct store file (judged by the filename), which means the content of the store file is not correct. I thought maybe it had to do with ignoring the version counts in the ColumnTrackers, but that does not appear to be the problem.
... going to have to shelve this for a bit to work on some other stuff.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157673#comment-13157673 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Thanks Jon.
v3 passes all tests (see above), I'll do a bit more sanity checking, and commit this some time tomorrow.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157362#comment-13157362 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Looking pretty good...
{noformat}
Results :
Failed tests: testClosing(org.apache.hadoop.hbase.client.TestHCM)
testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): expected:<1> but was:<0>
Tests run: 1065, Failures: 2, Errors: 0, Skipped: 7
{noformat}
testClosing is unrelated.
And a 2nd run for for distributed log splitting:
{noformat}
Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.653 sec
Results :
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Jonathan Hsieh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157653#comment-13157653 ]
Jonathan Hsieh commented on HBASE-4838:
---------------------------------------
@Lars:
+1 lgtm.
I didn't do a deep code review but I applied v3 and tested the TestAcidGuarantees ran it 20 times, and also ran the failures enumerated in HBASE-2856 they all pass. (Wow, the diff between v1 and v3 is pretty subtle.)
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157625#comment-13157625 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
So everybody's fine if I committed this now? Or should I wait until Monday to get more folks to look at it? Personally, I think this can go in.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157345#comment-13157345 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Finally found the problem!!
It was indeed a problem in HalfStoreFileReader (as Todd and Stack have suggested all along). HalfStoreFileReader did not have a getScanner(final boolean cacheBlocks, final boolean pread, final boolean isCompaction) method, and hence the super method was called returning a normal reader.
bq. Reference.java and HalfStoreFileReader.java are identical between 0.92 and trunk
I don't know what I was comparing here (must have been late in the night), but they are different and that difference was exactly the problem. Could have saved myself about 16 hours of debugging.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156357#comment-13156357 ]
stack commented on HBASE-4838:
------------------------------
Yeah, look see if TRUNK has a fix in Reference or HalfStoreFileReader.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4838:
---------------------------------
Summary: Port 2856 (TestAcidGuarantee is failing) to 0.92 (was: Port 2856 (TestAcidGuarantees is failing) to 0.92)
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157000#comment-13157000 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
I pinpointed the difference to the compactions of the daughters (again with just 2 keys):
in 0.92 (with this patch) I see this for the 1st daughter region (which is compacted last):
{noformat}
2011-11-24 22:08:51,324 INFO [RegionServer:2;localhost,42385,1322201325234-smallCompactions-1322201331230] regionserver.HRegion(1012): Starting compaction on testFamily in region testFilterAcrossMutlipleRegions,,1322201330936.0db66f8aabdf138dbbcf6c04f857c284.
2011-11-24 22:08:51,332 INFO [RegionServer:2;localhost,42385,1322201325234-smallCompactions-1322201331230] regionserver.Store(725): Starting compaction of 1 file(s) in testFamily of testFilterAcrossMutlipleRegions,,1322201330936.0db66f8aabdf138dbbcf6c04f857c284. into tmpdir=hdfs://localhost:52206/user/lars/testFilterAcrossMutlipleRegions/0db66f8aabdf138dbbcf6c04f857c284/.tmp, seqid=3, totalSize=662.0
2011-11-24 22:08:51,333 DEBUG [RegionServer:2;localhost,42385,1322201325234-smallCompactions-1322201331230] regionserver.Store(1174): Compacting hdfs://localhost:52206/user/lars/testFilterAcrossMutlipleRegions/0db66f8aabdf138dbbcf6c04f857c284/testFamily/85a0a11b15a248c69e09e44e0e9e052e.4e293f99103a49243c16eb104996554b-hdfs://localhost:52206/user/lars/testFilterAcrossMutlipleRegions/4e293f99103a49243c16eb104996554b/testFamily/85a0a11b15a248c69e09e44e0e9e052e-bottom, keycount=2, bloomtype=NONE, size=662.0
2011-11-24 22:08:51,388 INFO [RegionServer:2;localhost,42385,1322201325234-smallCompactions-1322201331230] regionserver.Store(1322): Renaming compacted file at hdfs://localhost:52206/user/lars/testFilterAcrossMutlipleRegions/0db66f8aabdf138dbbcf6c04f857c284/.tmp/7e7f4acb121e4696bd3c7d64e26a66b9 to hdfs://localhost:52206/user/lars/testFilterAcrossMutlipleRegions/0db66f8aabdf138dbbcf6c04f857c284/testFamily/7e7f4acb121e4696bd3c7d64e26a66b9
2011-11-24 22:08:51,402 INFO [RegionServer:2;localhost,42385,1322201325234-smallCompactions-1322201331230] regionserver.Store(746): Completed major compaction of 1 file(s) in testFamily of testFilterAcrossMutlipleRegions,,1322201330936.0db66f8aabdf138dbbcf6c04f857c284. into 7e7f4acb121e4696bd3c7d64e26a66b9, size=662.0; total size for store is 662.0
{noformat}
in trunk I see this for the 1st daughter region:
{noformat}
2011-11-24 22:15:18,205 INFO [RegionServer:0;localhost,46427,1322201712357-smallCompactions-1322201718071] regionserver.HRegion(1097): Starting compaction on testFamily in region testFilterAcrossMutlipleRegions,,1322201717807.2bdeac6934712efdd694ec44ae48d1b2.
2011-11-24 22:15:18,206 INFO [RegionServer:0;localhost,46427,1322201712357-smallCompactions-1322201718071] regionserver.Store(797): Starting compaction of 1 file(s) in testFamily of testFilterAcrossMutlipleRegions,,1322201717807.2bdeac6934712efdd694ec44ae48d1b2. into tmpdir=hdfs://localhost:37213/user/lars/testFilterAcrossMutlipleRegions/2bdeac6934712efdd694ec44ae48d1b2/.tmp, seqid=3, totalSize=718.0
2011-11-24 22:15:18,206 DEBUG [RegionServer:0;localhost,46427,1322201712357-smallCompactions-1322201718071] regionserver.Store(1255): Compacting hdfs://localhost:37213/user/lars/testFilterAcrossMutlipleRegions/2bdeac6934712efdd694ec44ae48d1b2/testFamily/64908313825b4c0599b86c26b33797e3.215be88f57f1ca63b6ead035b39c4d2e-hdfs://localhost:37213/user/lars/testFilterAcrossMutlipleRegions/215be88f57f1ca63b6ead035b39c4d2e/testFamily/64908313825b4c0599b86c26b33797e3-bottom, keycount=2, bloomtype=NONE, size=718.0
2011-11-24 22:15:18,211 INFO [RegionServer:0;localhost,46427,1322201712357-smallCompactions-1322201718071] regionserver.Store(818): Completed major compaction of 1 file(s) in testFamily of testFilterAcrossMutlipleRegions,,1322201717807.2bdeac6934712efdd694ec44ae48d1b2. into none, size=none; total size for store is 0.0
{noformat}
The keys in both cases are aaa and aab and the split key is aaa, so the 1st region (''-'aaa') should indeed be empty after compaction. In trunk it is correctly compacted to an empty file.
In 0.92 it somehow wrote out the entire file again (so the keys are found in the store files for both regions).
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4838) Port 2856
(TestAcidGuarantee is failing) to 0.92
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157345#comment-13157345 ]
Lars Hofhansl edited comment on HBASE-4838 at 11/26/11 12:30 AM:
-----------------------------------------------------------------
Finally found the problem!!
It was indeed a problem in HalfStoreFileReader (as Todd and Stack have suggested all along). HalfStoreFileReader did not have a getScanner(final boolean cacheBlocks, final boolean pread, final boolean isCompaction) method, and hence the super method was called, returning a "normal" ScannerV2, instead of the inner HFileScanner instance.
bq. Reference.java and HalfStoreFileReader.java are identical between 0.92 and trunk
I don't know what I was comparing here (must have been late in the night), but they are different and that difference was exactly the problem. Could have saved myself about 16 hours of debugging.
was (Author: lhofhansl):
Finally found the problem!!
It was indeed a problem in HalfStoreFileReader (as Todd and Stack have suggested all along). HalfStoreFileReader did not have a getScanner(final boolean cacheBlocks, final boolean pread, final boolean isCompaction) method, and hence the super method was called returning a normal reader.
bq. Reference.java and HalfStoreFileReader.java are identical between 0.92 and trunk
I don't know what I was comparing here (must have been late in the night), but they are different and that difference was exactly the problem. Could have saved myself about 16 hours of debugging.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is
failing) to 0.92
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157349#comment-13157349 ]
Lars Hofhansl commented on HBASE-4838:
--------------------------------------
Running all tests now.
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> ------------------------------------------------
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
> Issue Type: Sub-task
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this not trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira