[jira] [Created] (NIFI-15321) Add new execution mode “All Nodes Except Primary” to prevent Primary node overload in large-scale ListSFTP deployments

["orhanguvenc (Jira)" <ji...@apache.org>]

orhanguvenc created NIFI-15321:
----------------------------------

             Summary: Add new execution mode “All Nodes Except Primary” to prevent Primary node overload in large-scale ListSFTP deployments
                 Key: NIFI-15321
                 URL: https://issues.apache.org/jira/browse/NIFI-15321
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
    Affects Versions: 2.2.0
         Environment: Apache NiFi 2.2.0 running on Kubernetes/OpenShift, 5-node NiFi cluster. Each NiFi pod is allocated 20 CPU cores, 100 GB RAM, 32 GB JVM heap, and configured with 80 Timer Driven threads. Approximately 720 ListSFTP processors operate simultaneously due to 120 integrations × 6 remote SFTP hosts.
            Reporter: orhanguvenc


h2. *Problem Overview*

In Kubernetes/OpenShift-based NiFi clusters, Primary-only processors such as *ListSFTP* cause severe load concentration on the Primary node.
In our environment, we run:
 * 5 NiFi pods

 * Each pod: *20 CPU cores, 100 GB RAM, 32 GB heap, 80 Timer Driven threads*

 * *~720 ListSFTP processors* (120 integrations × 6 remote SFTP hosts)

Because ListSFTP must run with {*}Execution = Primary Node{*}, all 720 ListSFTP processors execute exclusively on whichever pod becomes Primary.
This saturates the Primary’s thread pool and CPU capacity, causing:
 * Thread starvation on Primary

 * Downstream processors (FetchSFTP,UpdateAttribute, CompressContent, DetectDublicate, CustomProcessor etc.) on the Primary node to stop making progress

 * FlowFiles assigned to Primary via load-balanced connections to remain stuck in queues

 * Cluster-wide throughput collapse and processing imbalance

This is {*}not solvable by adding hardware{*}, because:
 * Primary election is dynamic in Kubernetes

 * Any pod can become Primary

 * Every pod must be sized for the worst-case ListSFTP load

 * Scheduling limitations—not hardware—cause the bottleneck

----
h2. *Requested Improvement*
h3. *Add a new execution mode: “All Nodes Except Primary”*

This mode would allow heavy downstream processors to run on {_}all cluster nodes except the Primary{_}.
Primary would continue running Primary-only processors (e.g., ListSFTP), without being required to handle the high CPU/thread load of downstream processing.
h3. *Benefits*
 * Prevents Primary node overload

 * Ensures downstream processors never run on a saturated Primary

 * Eliminates queue imbalance caused by Primary starvation

 * Improves cluster-wide throughput and stability

 * Fully compatible with Kubernetes/OpenShift ephemeral scheduling

 * Reduces the need for overprovisioning (20 CPUs / 100 GB RAM per pod becomes unnecessary)

----
h2. *Alternative Enhancements (Secondary options)*
 # *Dedicated thread pool for Primary-only processors*
Prevents them from starving general processor scheduling.

 # *Improved workload isolation for listing processors*
Allows Primary to handle coordination work without blocking dataflow execution.

However, the *minimum impactful change* is adding:
h3. *Execution = All Nodes Except Primary*
----
h2. *Conclusion*

Large ingestion architectures (hundreds of ListSFTP processors) require improved scheduling primitives to keep clusters stable.
Introducing *“All Nodes Except Primary”* would allow NiFi to scale horizontally and reliably in modern Kubernetes environments and prevent Primary node overload that occurs with stateful listing processors.

We will gladly provide thread dumps, scheduling traces, and cluster performance metrics if needed for development or validation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)