[jira] [Created] (KUDU-3453) Fine-grained anchoring for WAL segments during tablet copying

["Alexey Serbin (Jira)" <ji...@apache.org>]

Alexey Serbin created KUDU-3453:
-----------------------------------

             Summary: Fine-grained anchoring for WAL segments during tablet copying
                 Key: KUDU-3453
                 URL: https://issues.apache.org/jira/browse/KUDU-3453
             Project: Kudu
          Issue Type: Improvement
          Components: tablet, tserver
            Reporter: Alexey Serbin


Tablet copying is a provision to implement the process of automatic tablet re-replication in Kudu.  When the system catalog (Kudu master) detects that a tablet replica is no longer available, it automatically re-replicates a tablet to a destination tablet server using another healthy tablet replica in the cluster as the source.

When copying a tablet from one tablet server to another, the source tablet copying session "anchors" WAL segments to be transfered to the destination server, so they are not GC-ed by the tablet maintenance operation when they are no longer needed locally, but the tablet copy session is still in progress.

The anchored WAL segments are releases all at once when the tablet copying session completes with success of failure.  However, there might be long running tablet copying sessions, and with high data ingest rate, the source tablet replica might accumulate huge amount of WAL data which isn't relevant at both the source and the destination server.

To prevent accumulation of WAL data for long-running tablet copying sessions, it's necessary to update the WAL anchors in a more granular manner, e.g. un-anchor a segment once it has been successfully copied and persisted by the client tablet copying session.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)