[jira] [Created] (HBASE-23205) Update currently replicating log position correctly.

["Jeongdae Kim (Jira)" <ji...@apache.org>]

Jeongdae Kim created HBASE-23205:
------------------------------------

             Summary: Update currently replicating log position correctly.
                 Key: HBASE-23205
                 URL: https://issues.apache.org/jira/browse/HBASE-23205
             Project: HBase
          Issue Type: Bug
    Affects Versions: 1.4.10, 1.5.0, 1.4.11
            Reporter: Jeongdae Kim
            Assignee: Jeongdae Kim


We observed a lot of old WALs were not removed from archives and their corresponding replication queues, while testing with 1.4.10.
stacked old WALs are empty or have no entries to be replicated (not in replication table_cfs)
 
As described in HBASE-22784, {color:#333333}if no entries to be replicated are appended to WALs, log position will never be updated. As a consequence, all WALs won’t be removed. this issue{color} happened since HBASE-15995.
 
{color:#333333}I think old WALs would not be stacked with {color}HBASE-22784. but, it still have something to be fixed as below * log position could be updated wrongly, when log rolled, because lastWalPath of batches might not point to WAL currently being read.

 
For example,  after last entries to be added in batch was read from P1 position in the WAL W1
and then WAL rolled, and reader read until it reaches the end of old wals and continue reading entries from new WAL W2, and then it reached batch size. current read position for W2 is P2. In this case, the batch being passed to a shipper have walPath W1 and P2, so shipper will try to update position P2 for W1. it may result in data inconsistency in recovery case or update failure to zookeeper (znode could not exist by previous log position updates, i guess this case is the same case asHBASE-23169?)
  * log position could be not updated or updated to wrong position by pendingShipment flag introduced from HBASE-22784

 
In shipper thread, it would not beguaranteed to update log position always by setting pendingShipment to false.
If  reader set the flag to true, right after shipper set it to false during {color:#24292e}updateLogPosition(), shipper won’t update log position.{color}
On the other hand, while reader read filtered entries, If shipper set to false reader will update log position to current read position. it may lose data in recovery case.
  * a lot of log position updates could be happened, when most of WAL entries are filtered by TableCfWALEntryFilter.

 
I think it would be better to reduce the number of log updates in that case, because # 
zookeeper writes are more expensive operations than reads.(since writes involve synchronizing the state of all servers),
 # 
even if read position was not updated, it would be harmless because all entries will be filtered out again in recovery process.

I think it's enough to update log position only when wal rolled in that case. (to cleanup old wals)
 
In addition, During this work, i found a minor bug which is updating replication buffer size wrongly by decreasing total buffer size with the size of bulk loaded files.
I’d like to fix it, if it’s ok.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)