[jira] [Commented] (HBASE-22075) Potential data loss when MOB compaction fails

["Karthik Palanisamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832954#comment-16832954 ] 

Karthik Palanisamy commented on HBASE-22075:
--------------------------------------------

[~elserj] Tested [~vrodionov] patch. This fix handled MOB data well safe. No data loss is noticed during MOB compaction failure, RS crash, IO issue, and any infra issues.

But majorly race condition is noticed w/ or w/o patch in the MOB. We see data loss again during MOB-compaction and Major-compaction while those are running together. 

As [~vrodionov] already mentioned, there will be a race condition in this case.  I think he already working on a new patch.

I have attached a small repro code (ReproMOBDataLoss.java) for this race condition. This is an aggressive test.  Test duration is nearly an hour.
 # Settings: Region Size 200 MB,  Flush threshold 800 KB.
 # Insert 10 Million records
 # MOB Compaction and Archiver
        a) Trigger MOB Compaction (every 2 minutes)
        b) Trigger major compaction (every 2 minutes)
        c) Trigger archive cleaner (every 3 minutes)
 # Validate MOB data after complete data load.

I ran this repro code on branch-2.2. The issue is reproduced. 

Also, ran this repro code after disabling MOB compaction. No data loss is noticed.

> Potential data loss when MOB compaction fails
> ---------------------------------------------
>
>                 Key: HBASE-22075
>                 URL: https://issues.apache.org/jira/browse/HBASE-22075
>             Project: HBase
>          Issue Type: Bug
>          Components: mob
>    Affects Versions: 2.1.0, 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4, 2.1.3
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>            Priority: Critical
>              Labels: compaction, mob
>             Fix For: 2.0.6, 2.1.5, 2.2.1
>
>         Attachments: HBASE-22075-v1.patch, HBASE-22075-v2.patch, ReproMOBDataLoss.java
>
>
> When MOB compaction fails during last step (bulk load of a newly created reference file) there is a high chance of a data loss due to partially loaded reference file, cells of which refer to (now) non-existent MOB file. The newly created MOB file is deleted automatically in case of a MOB compaction failure, but some cells with the references to this file might be loaded to HBase. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)