[jira] Commented: (HBASE-3235) Intermittent incrementColumnValue failure in TestHRegion

["Gary Helmling (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932265#action_12932265 ] 

Gary Helmling commented on HBASE-3235:
--------------------------------------

Some further info on this.  I added logging of the contents of store.memstore.kvset when kvset.size() > 1 in testIncrementColumnValue_UpdatingInPlace(), like so:

{code}
    long value = 1L;
    long amount = 3L;

    Put put = new Put(row);
    put.add(fam1, qual1, Bytes.toBytes(value));
    region.put(put);

    long result = region.incrementColumnValue(row, fam1, qual1, amount, true);

    assertEquals(value+amount, result);

    Store store = region.getStore(fam1);
    // ICV removes any extra values floating around in there.
    if (store.memstore.kvset.size() > 1) {
      for (KeyValue kv : store.memstore.kvset) {
        LOG.debug("memstore.kvset: row="+Bytes.toString(kv.getRow()) +
            " fam="+Bytes.toString(kv.getFamily())+" col="+Bytes.toString(kv.getQualifier())+
            " val="+Bytes.toLong(kv.getValue())+" ts="+kv.getTimestamp()+" memstore_ts="+kv.getMemstoreTS());
      }
    }
    assertEquals(1, store.memstore.kvset.size());
    assertTrue(store.memstore.snapshot.isEmpty());
{code}

With this in place, I get the following output for the failure case:

{noformat}
2010-11-15 15:36:24,242 DEBUG [main] regionserver.TestHRegion(1891): memstore.kvset: row=rowA fam=colfamily1 col=qual1 val=1 ts=1289864184241 memstore_ts=1
2010-11-15 15:36:24,242 DEBUG [main] regionserver.TestHRegion(1891): memstore.kvset: row=rowA fam=colfamily1 col=qual1 val=4 ts=1289864184241 memstore_ts=0
{noformat}

So it seems like it's only happening when the timestamps for the initial put and the ICV are identical.  In this case, the put gets memstoreTS=1 (from RWCC.getWriteNumber()), but the ICV memstoreTS=0 in MemStore.upsert(List).

So, in MemStore.upsert(KeyValue), when we call kvset.tailSet(kv), we're missing the initial put, because KeyValue.KV_COMPARATOR is evaluating it as less than the ICV put, by inverting the memstoreTS comparison.

Given that that's the cause, what is the correct fix?  Should the ICV put get using RWCC.getWriteNumber() as well?  Or should we be obtaining the tailset without regards to the memstoreTS?

> Intermittent incrementColumnValue failure in TestHRegion
> --------------------------------------------------------
>
>                 Key: HBASE-3235
>                 URL: https://issues.apache.org/jira/browse/HBASE-3235
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.0
>            Reporter: Gary Helmling
>
> I first saw this in a Hudson build, but can reproduce locally with enough test runs (5-10 times):
> {noformat}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.hbase.regionserver.TestHRegion
> -------------------------------------------------------------------------------
> Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.413 sec <<< FAILURE!
> testIncrementColumnValue_UpdatingInPlace(org.apache.hadoop.hbase.regionserver.TestHRegion)  Time elapsed: 0.079 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<1> but was:<2>
>         at junit.framework.Assert.fail(Assert.java:47)
>         at junit.framework.Assert.failNotEquals(Assert.java:283)
>         at junit.framework.Assert.assertEquals(Assert.java:64)
>         at junit.framework.Assert.assertEquals(Assert.java:195)
>         at junit.framework.Assert.assertEquals(Assert.java:201)
>         at org.apache.hadoop.hbase.regionserver.TestHRegion.testIncrementColumnValue_UpdatingInPlace(TestHRegion.java:1889)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> Alternately, the failure can also show up in testIncrementColumnValue_UpdatingInPlace_Negative():
> {noformat}
> testIncrementColumnValue_UpdatingInPlace_Negative(org.apache.hadoop.hbase.regionserver.TestHRegion)  Time elapsed: 0.03 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<3>
>         at junit.framework.Assert.fail(Assert.java:47)
>         at junit.framework.Assert.failNotEquals(Assert.java:283)
>         at junit.framework.Assert.assertEquals(Assert.java:64)
>         at junit.framework.Assert.assertEquals(Assert.java:130)
>         at junit.framework.Assert.assertEquals(Assert.java:136)
>         at
> org.apache.hadoop.hbase.regionserver.TestHRegion.assertICV(TestHRegion.java:2081)
>         at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testIncrementColumnValue_UpdatingInPlace_Negative(TestHRegion.java:1990)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.