[jira] [Commented] (HBASE-27871) Meta replication stuck forever if wal it's still reading gets rolled and deleted

["Hudson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-27871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729930#comment-17729930 ] 

Hudson commented on HBASE-27871:
--------------------------------

Results for branch branch-2
	[build #827 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/827/]: (x) *{color:red}-1 overall{color}*
----
details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/827/General_20Nightly_20Build_20Report/]


(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/827/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/827/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/827/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Meta replication stuck forever if wal it's still reading gets rolled and deleted
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-27871
>                 URL: https://issues.apache.org/jira/browse/HBASE-27871
>             Project: HBase
>          Issue Type: Bug
>          Components: meta replicas
>    Affects Versions: 2.6.0, 2.4.17, 2.5.4
>            Reporter: Wellington Chevreuil
>            Assignee: Wellington Chevreuil
>            Priority: Major
>             Fix For: 2.6.0
>
>
> This affects branch-2 based releases only (in master, HBASE-26416 refactored region replication to not rely on the replication framework anymore).
> Per the original [meta region replicas design|https://docs.google.com/document/d/1jJWVc-idHhhgL4KDRpjMsQJKCl_NRaCLGiH3Wqwd3O8/edit], we use most of the replication framework for communicating changes in the primary replica back to the secondary ones, but we skip storing the queue state in ZK. In the event of a region replication crash, we should let the related replication source thread be interrupted, so that 
> RegionReplicaReplicationEndpoint would set a new source from the scratch and make sure to update the secondary replicas.
>  
> We have run into a situation in one of our customers' cluster where the region replica source faced a long lag (probably because the RSes hosting the secondary replicas were busy and slower in processing the region replication entries), so that the current wal got rolled and eventually deleted whilst the replication source reader was still referring it. In such cases, ReplicationSourceReader only sees the IOException and keeps retrying the read indefinitely, but since the file is now gone, it will just get stuck there forever. In the particular case of FNFE (which I believe would only happen for region replication), we should just raise an exception and let RegionReplicaReplicationEndpoint handle it to reset the region replication source.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)