You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Vinayak Hegde (Jira)" <ji...@apache.org> on 2025/04/15 07:37:00 UTC
[jira] [Commented] (HBASE-29220) Track the Age/Timestamp of the Last Successfully Backed-Up WAL Entry in Continuous Backup Replication Endpoint
[ https://issues.apache.org/jira/browse/HBASE-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17944640#comment-17944640 ]
Vinayak Hegde commented on HBASE-29220:
---------------------------------------
h3. Overview
We are implementing a mechanism to compute the replication checkpoint timestamp used in continuous backups. This timestamp helps ensure that any data restored during Point-in-Time Restore (PITR) or incremental backups reflects a consistent and fully replicated state — i.e., no WAL entries prior to the checkpoint are missing from the target system.
This logic relies heavily on two core components of HBase's replication framework:
h3. What are WAL Replication Queues and the Replication Marker Chore?
1. WAL Replication Queues:
Each region server in HBase maintains a list of WALs that need to be replicated to a peer cluster. These are stored in ZooKeeper and are managed by the replication framework. The WALs in the queue represent the unreplicated or partially replicated data. The start timestamp of each WAL indicates the earliest possible point at which replication work may still be pending for that server.
2. Replication Marker Chore:
This is a background process that periodically scans the replication status and updates a marker table in HBase. For each region server, it writes the latest timestamp up to which data is known to be successfully replicated. This provides a higher-level summary of replication progress, independent of WAL structures.
h3. How We Use These in Our Approach
We combine both the WAL replication queues and the replication marker chore to calculate the replication checkpoint:
* From WAL Queues:
We extract the start timestamps of the oldest WALs for each region server. These serve as a conservative lower bound — replication cannot be considered complete before these points.
* From Marker Chore:
We read the last replicated timestamp per region server from the marker table. These can show more advanced replication progress, especially if WALs haven't been rolled recently.
* Combining the Two:
For each region server:
** If it has both WAL and marker entries, and the marker timestamp is more recent, we use the marker (replication has moved beyond the WAL's start).
** If a marker exists but the region server no longer has WALs, we discard the marker — the server might be decommissioned or no longer relevant.
** If the marker chore is disabled, we fallback to WAL-only approach.
Finally, we compute the minimum of all relevant timestamps to get the checkpoint — ensuring we don’t skip over any unreplicated data.
h3. Why Use Both?
Relying on only one of the two sources creates problems:
* WAL-Only Issues:
If a WAL is long-lived (not rolled for hours), the start timestamp remains stale, even if replication has actually progressed far ahead.
_Example:_ WAL {{WAL-001}} started at {{{}T1000{}}}. It’s still in the queue, but data has already been replicated up to {{{}T6000{}}}. If we only rely on WALs, the checkpoint gets stuck at {{{}T1000{}}}, unnecessarily delaying PITR.
* Marker-Only Issues:
The marker chore may leave stale entries for decommissioned or reassigned servers.
_Example:_ {{rs1}} was handling WALs at {{{}T10{}}}. It gets decommissioned. Now {{rs2}} takes over the same data and has replicated it up to {{{}T20{}}}. However, {{{}rs1{}}}'s marker still says {{{}T10{}}}. If we blindly include that stale entry, the checkpoint incorrectly remains at {{{}T10{}}}, even though the system has moved forward.
So rather than being too aggressive, we are actually overly cautious — and intentionally so — to ensure correctness even if it means conservative (i.e., slower-moving) checkpoints.
That’s why we:
* Use marker timestamps only if the server still has WALs (i.e., it's active),
* Discard markers from servers not present in WAL queues (assuming they're no longer relevant),
* Always take the minimum across valid sources to ensure no unreplicated data is skipped.
> Track the Age/Timestamp of the Last Successfully Backed-Up WAL Entry in Continuous Backup Replication Endpoint
> --------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-29220
> URL: https://issues.apache.org/jira/browse/HBASE-29220
> Project: HBase
> Issue Type: Task
> Components: backup&restore
> Reporter: Vinayak Hegde
> Priority: Major
>
> We use HBase’s replication framework for Continuous Backup through {{{}ContinuousBackupReplicationEndpoint{}}}. This replicates WAL entries to the backup location, which are then used for Point-In-Time Recovery (PITR) and Incremental Backup (an optimization technique that collects WALs and generates HFiles for faster recovery).
> However, the {{ReplicationEndpoint}} can lag behind in time.
> For example, if replication is one hour behind, {{ContinuousBackupReplicationEndpoint}} will currently be writing WAL entries that are one hour old. This means that if a user requests a PITR for the current time or attempts an incremental backup, they will miss that one hour of data.
> To prevent this, we need to ensure that users can only request data that has been fully backed up. Therefore, we must track the timestamp of the last successfully backed-up WAL entry:
> * For PITR: Users should only be allowed to restore to a point before this timestamp.
> * For Incremental Backup: The incremental backup process should store this timestamp as the backup time to maintain data consistency.
> This ensures data integrity and prevents users from requesting backups that include unprocessed WAL entries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)