[jira] [Commented] (HBASE-26849) NPE caused by WAL Compression and Replication

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577372#comment-17577372 ] 

Bryan Beaudreault commented on HBASE-26849:
-------------------------------------------

I will point out that our refguide docs show a warning regarding this issue: [https://hbase.apache.org/book.html#_cluster_replication]
{quote}At present, there is compatibility problem if Replication and WAL Compression are used together. If you need to use Replication, it is recommended to set the {{hbase.regionserver.wal.enablecompression}} property to {{{}false{}}}. See (HBASE-26849) for details.
{quote}
There was also a notice on the dev/user list. The affects versions here show 2.x affected. We disabled WAL compression per those notices, and I'm sure are not alone.

I don't know enough about the problem here to say, but at the surface it seems like this affects active branches. Anyone know better?

 

> NPE caused by WAL Compression and Replication
> ---------------------------------------------
>
>                 Key: HBASE-26849
>                 URL: https://issues.apache.org/jira/browse/HBASE-26849
>             Project: HBase
>          Issue Type: Bug
>          Components: Replication, wal
>    Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.11
>            Reporter: tianhang tang
>            Assignee: tianhang tang
>            Priority: Critical
>         Attachments: image-2022-03-16-14-25-49-276.png, image-2022-03-16-14-30-15-247.png
>
>
> My cluster uses HBase 1.4.12, opened WAL compression and replication.
> I could found replication sizeOfLogQueue backlog, and after some debugs, found the NPE throwed by [https://github.com/apache/hbase/blob/branch-1/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/LRUDictionary.java#L109:]
> !image-2022-03-16-14-25-49-276.png!
>  
> The root cause for this problem is:
> WALEntryStream#checkAllBytesParsed:
> !image-2022-03-16-14-30-15-247.png!
> resetReader does not create a new reader, the original CompressionContext and the dict in it will still be retained.
> However, at this time, the position is reset to 0, which means that the HLog needs to be read from the beginning, but the cache that has not been cleared is still used, so there will be problems: the same data has already in the LRUCache, and it will be directly added to the cache again.
> Recreate a new reader here, the problem is solved.
> I will open a PR later. But, there are some other places in the current code to resetReader or seekOnFs. I guess these codes doesn't take into account the wal compression case at all...
>  
> In theory, as long as the file is read again, the LRUCache should also be rolled back, otherwise there will be inconsistent behavior of READ and WRITE links.
> But the position can be roll back to any intermediate position at will, but LRUCache can't...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)