[jira] [Updated] (HBASE-11842) Integration test for async wal replication to secondary regions

["Enis Soztutar (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-11842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Enis Soztutar updated HBASE-11842:
----------------------------------
    Attachment: hbase-11842_v1.patch

Attaching a patch for an integration test for the async wal replicas feature. This single test covers most of the testing HBASE-10070 Phase 2 items including Async WAL replication, memory pressure handling, and failover handling. Using this test allowed us to uncover a couple of fixes that are already fixed in the patches, or upcoming in the follow up patches (HBASE-12561, HBASE-11332, HBASE-11908, etc). 

The test runs LoadTestTool with N writes and M reader threads. The logic for writer and readers (key distribution, verification, etc) is used as it is. The test initializes a table with replicas and also assumes that async wal replication is enabled in the cluster (via configuration). 

The writer threads pick up keys to write and they perform the writes to the table via primary regions. The expectation is that async wal replication will replicate the modified keys shortly after to the region replicas. There is a queue between writers and readers in LTT where the writers pick up keys to write, and when the write is complete, they put the key to the queue so that readers pick it up from there. This test simply changes the queue to be a delay queue, which introduces a configurable amount of delay (for example 30 sec) between a writer thread putting the key to the queue, and making it available to a reader thread. This means that the readers are reading keys written 30 sec or more before. The readers in the test read the keys from a region replica an verify. If the async wal replication is broken, or there is data loss between replicas, the test will fail. The test can also run chaos monkey. 


 

> Integration test for async wal replication to secondary regions
> ---------------------------------------------------------------
>
>                 Key: HBASE-11842
>                 URL: https://issues.apache.org/jira/browse/HBASE-11842
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0
>
>         Attachments: hbase-11842_v1.patch
>
>
> We should add an integration test to cover async wal replication to secondary regions. 
> Async wal replication to region replicas has been committed (HBASE-11568) and flush /compaction handling (HBASE-11569) and failover handling for secondary region replicas (HBASE-11580) is in the works.  
> The test can use LoadTestTool over a table with region replicas. We can write to the primary regions, and read from the secondary region replicas in reader threads to do the verification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)