[jira] [Updated] (NIFI-15570) Partial defragmentation of Content Repository via tail-claim truncation

["Pierre Villard (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pierre Villard updated NIFI-15570:
----------------------------------
    Fix Version/s:     (was: 2.9.0)

> Partial defragmentation of Content Repository via tail-claim truncation
> -----------------------------------------------------------------------
>
>                 Key: NIFI-15570
>                 URL: https://issues.apache.org/jira/browse/NIFI-15570
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> h3. Problem
> NiFi's FileSystemRepository uses a slab-allocation strategy for storing FlowFile content: multiple FlowFiles are written sequentially into a single ResourceClaim file on disk. This is efficient because it avoids the overhead of creating and deleting huge numbers of small files. However, it introduces a fragmentation problem.When any FlowFile still references a ResourceClaim, the entire file must be kept on disk — even if the vast majority of its bytes belong to FlowFiles that have already been removed. Consider a ResourceClaim that contains five ContentClaims of sizes 1 KB, 2 KB, 4 KB, 3 KB, and 1 GB. If only the 1 KB FlowFile remains, the full ~1 GB file stays on disk. At scale, this leads to disk exhaustion.A full defragmentation (rewriting live claims into new ResourceClaim files, updating all references, and deleting the originals) would be extremely complex and expensive. But it turns out we can solve the vast majority of the problem without it.
> h3. Key Insight
> With NiFi's slab allocation, there are three possible positions for a large ContentClaim within a ResourceClaim:
>  
>  
> {{<> = Small FlowFile}}
> [................] = Large FlowFile
>  
> 1. Beginning:   [................]<><><><><><><>
>  
> 2. Middle:      <><><><>[................]<><><>
>  
> 3. End:         <><><><><><><><>[................]
>  
> NiFi already prevents cases 1 and 2. The nifi.content.claim.max.appendable.size property (default: 50 KB) causes the repository to stop appending to a ResourceClaim once it exceeds that threshold. Since a "large" ContentClaim is by definition larger than this threshold, the act of writing it will push the ResourceClaim past the (soft) limit, causing the ResourceClaim to be closed for further appending. No additional ContentClaims can be written after the large one.This means a large ContentClaim can only ever appear at the tail of a ResourceClaim. And truncating a file from the tail requires no data movement — it is a single FileChannel.truncate() call.
> h3. Solution
> This change implements "partial defragmentation" by truncating ResourceClaim files from the tail when the last (large) ContentClaim is removed. The approach consists of several coordinated components:Marking truncation candidates at write time — When a ContentClaim is closed in FileSystemRepository, the repository checks whether it is both (a) large (exceeding a threshold) and (b) at a non-zero offset (i.e., not the only claim in the file). If both conditions hold, the claim is flagged as a truncation candidate via StandardContentClaim.setTruncationCandidate(true). If the claim is later cloned (claimant count incremented), the flag is cleared, since truncation is only safe when the claim has a single owner.Routing truncatable claims through the FlowFile Repository — When a FlowFile is deleted or its content is replaced, WriteAheadFlowFileRepository.updateContentClaims() checks if the released ContentClaim is a truncation candidate. If so (and the ResourceClaim itself is not already fully destructable), the claim is queued in claimsAwaitingTruncation. On the next WAL checkpoint or sync, these claims are drained to ResourceClaimManager.markTruncatable().Background truncation in FileSystemRepository — A scheduled TruncateClaims task periodically drains truncatable claims from the ResourceClaimManager. Before truncating, it checks whether truncation is active for the claim's container (archive must be cleared on the last cleanup run and disk usage must exceed the configured threshold). If conditions are met, the file is truncated to the claim's offset via FileChannel.truncate(). If conditions are not met, the claims are saved in a TruncationClaimManager and retried on subsequent runs, ensuring no truncation opportunity is lost.Recovery — On restart, WriteAheadFlowFileRepository.restoreFlowFiles() re-derives truncation eligibility by scanning all recovered FlowFiles, identifying large claims at non-zero offsets that are at the tail of their ResourceClaim and are not shared by multiple FlowFiles.
> h3. Example
> Before truncation — a 1 GB FlowFile was removed but the ResourceClaim persists because small FlowFiles still reference it:
>  
>  
>  
> {{ResourceClaim file (1,000,010 KB on disk):}}{{  [1 KB] [2 KB] [4 KB] [3 KB] [1,000,000 KB (removed)]}}{{After truncation — the file is truncated at the offset where the large claim began:}}
>  
> {{ResourceClaim file (10 KB on disk):}}{{  [1 KB] [2 KB] [4 KB] [3 KB]}}
>  
> The small FlowFiles remain fully readable. The 1 GB of wasted space is reclaimed instantly with a single syscall.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)