[jira] [Created] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys

["Mikhail Antonov (JIRA)" <ji...@apache.org>]

Mikhail Antonov created HBASE-16232:
---------------------------------------

             Summary: ITBLL fails on branch-1.3, now loosing actual keys
                 Key: HBASE-16232
                 URL: https://issues.apache.org/jira/browse/HBASE-16232
             Project: HBase
          Issue Type: Bug
          Components: dataloss, integration tests
    Affects Versions: 1.3.0
            Reporter: Mikhail Antonov
            Priority: Blocker


So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix for fake keys showing up in the scans) with increased number of regions per regionserver and seeing the following.

{quote} 
$Verify​$Counts	
REFERENCED	0	4,999,999,994	4,999,999,994
UNDEFINED	0	3	3
UNREFERENCED	0	3	3
{quote}

So we're loosing some keys. This time those aren't fake:

{quote}
undef	
\x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F	0	1	1
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3	0	1	1
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A	0	1	1
unref	
\x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF	0	1	1
\xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC	0	1	1
U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
{quote}

Re-running verify step with CM off still shows this issue. Search tool reports:
{quote}
Total
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3	5	0	5
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A	4	0	4
CELL_WITH_MISSING_ROW	15	0	15
{quote}

Will post more as I dig into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)