You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2026/10/09 21:18:00 UTC
[jira] [Updated] (PHOENIX-7977) Stage forwarded replication log files in a per-shard staging subdirectory before atomic rename
[ https://issues.apache.org/jira/browse/PHOENIX-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Kyle Purtell updated PHOENIX-7977:
-----------------------------------------
Fix Version/s: PHOENIX-7562-feature
5.4.0
> Stage forwarded replication log files in a per-shard staging subdirectory before atomic rename
> ----------------------------------------------------------------------------------------------
>
> Key: PHOENIX-7977
> URL: https://issues.apache.org/jira/browse/PHOENIX-7977
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Tanuj Khurana
> Assignee: Tanuj Khurana
> Priority: Major
> Fix For: PHOENIX-7562-feature, 5.4.0
>
>
> When a fallback (standby) cluster forwards a replication log file to the peer cluster, ReplicationLogDiscoveryForwarder.processFile() copied the bytes directly onto the file's final replay-eligible name <shard>/<ts>_<origin>.plog using FileUtil.copy(..., overwrite=false).
> FileUtil.copy creates the destination at that final name before close() completes. The standby replay consumer lists the shard directory on its own cadence, and can thereforepick up the still-open, half-written file. When it does, it force-recovers the HDFS lease on that file (single-writer model), which makes the forwarder's own close() fail with LeaseExpiredException.
> That exception propagates out of processFile() before the STORE_AND_FORWARD → SYNC_AND_FORWARD throughput check runs, so the HA group's transition back toward SYNC is delayed. The source file also remains in the out-progress directory and is retried, but each retry can re-lose the same race.
>
> Fix
> Publish forwarded files atomically instead of exposing them mid-copy:
> 1. Add ReplicationShardDirectoryManager.STAGING_SUB_DIRECTORY_NAME = ".staging" and getStagingPath(finalPath), which returns <shard>/.staging/<file-name>. The staged file keeps its real .plog name but lives one level down in a subdirectory.
> 2. The forwarder copies bytes into the staging path (overwrite=true, to reclaim any orphan left by a prior crashed attempt), then does a same-shard, same-FileSystem rename(staging, dst) to publish once the file is fully written.
> 3. On rename returning false: if dst already exists (a retry raced ahead of replay) delete the staging copy and treat it as delivered; otherwise throw so the source is retried. Delivery stays at-least-once, which is safe because replay is idempotent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)