[jira] [Resolved] (NIFI-16200) Improve resilience against large/numerous FlowFile attributes

["Mark Payne (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-16200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Payne resolved NIFI-16200.
-------------------------------
    Resolution: Duplicate

> Improve resilience against large/numerous FlowFile attributes
> -------------------------------------------------------------
>
>                 Key: NIFI-16200
>                 URL: https://issues.apache.org/jira/browse/NIFI-16200
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>
> h2. Motivation
> FlowFile attributes are intended to be small metadata (typically on the order of hundreds of characters). In practice, flows often place large payloads in attributes. Attributes are held in memory on each FlowFile and all attributes are rewritten to the FlowFile Repository on every update that includes any attribute change. Large attributes therefore increase write cost, inflate the FlowFile Repository write-ahead log (WAL) between checkpoints, and can exhaust JVM heap before today's count-based swap thresholds engage.
> Today, swap is driven by a globally configured FlowFile count ({{{}nifi.queue.swap.threshold{}}}, default 20,000) with a fixed batch size of 10,000 FlowFiles written per swap file.
> FlowFile Repository checkpointing is time-based (default 20 seconds). Under heavy attribute write load, the active journal can grow large enough to exhaust disk before the next scheduled checkpoint.
> As a result, a huge number of attributes, or very large attributes, cause issues with both NiFi's JVM heap as well as disk contention contention and storage space exhaustion.
> h2. Proposed Solution
> Two related changes:
>  # *Size-triggered FlowFile Repository checkpointing.* Keep the existing time-based checkpoint interval. Add a configurable maximum size for the active WAL journal. When that size is reached, force a checkpoint instead of waiting for the interval, then begin a new journal and discard the prior snapshot/journals as today.
>  # *Per-queue, heap-size-based swapping.* Replace globally configured count-based swap thresholds with a per-connection budget for estimated in-heap FlowFile metadata (record overhead plus attribute key/value lengths). Default global budget: 10 MB of metadata in the active queue before overflow begins. Overflow uses the existing secondary in-heap queue; when that queue holds about 1 MB of estimated metadata, write a swap file and drop those FlowFiles from heap. Each connection may override the budget (same configuration pattern as backpressure object/data-size thresholds). A global default remains in {{{}nifi.properties{}}}.
> h2. NiFi API Changes
>  * {{VersionedConnection}} gains a field for the per-connection metadata heap swap budget (data-size string, consistent with backpressure data-size fields).
> h2. REST API Changes
>  * {{ConnectionDTO}} (and related create/update paths) expose the per-connection metadata heap swap budget.
> h2. Framework Changes
>  * Estimate FlowFile metadata heap size from the FlowFile record structure and attribute key/value lengths.
>  * Drive active-queue overflow and swap-file batching from estimated metadata bytes in addition to FlowFile counts (including clustered queue partitions).
>  * FlowFile Repository: force checkpoint when the active journal reaches the configured size threshold, in addition to the existing time-based schedule and the existing post-recovery checkpoint on startup.
> h2. UI Changes
>  * Create Connection and Edit Connection gain a field for the metadata heap swap budget (alongside FlowFile expiration and backpressure).
> h2. nifi.properties Configuration
>  * New property for the default per-queue metadata heap swap budget (default {{{}10 MB{}}}). This supersedes {{nifi.queue.swap.threshold}} as the primary swap control; the count-based property is removed.
>  * New property for the maximum FlowFile Repository WAL journal size that forces a checkpoint.
> This set of changes provide significant protection to NiFi's JVM heap, as well as storage constraints. Additionally, it does so in a way that provides queue-specific overrides so that if there are specific queues that do need to support larger heap utilization without swapping, that can be accommodated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)