[jira] [Resolved] (NIP-44) Allow Stateless groups to keep FlowFile content in memory

["Mark Payne (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIP-44?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Payne resolved NIP-44.
---------------------------
    Resolution: Done

Vote passed; NIP ratified

> Allow Stateless groups to keep FlowFile content in memory
> ---------------------------------------------------------
>
>                 Key: NIP-44
>                 URL: https://issues.apache.org/jira/browse/NIP-44
>             Project: NiFi Improvement Proposal
>          Issue Type: New Feature
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>
> h1. Background
> When Stateless NiFi is used directly via cli or embedded, it is typically run with an in-memory (on-heap) Content Repository. This makes a lot of sense because it's much more performance than writing to disk and it generally 'owns' the full JVM. No risk of using a lot of heap and causing completely separate flows to be memory constrained.
> However, when we introduced Stateless Process Groups in 'NiFi proper' we did not set it up this way. We set it up to write content directly to the configured Content Repository, even though the content claim will have no references and will be destroyed upon restart of NiFi. We did this out of concerns for stability. If you have a single Stateless group that buffers a couple GB of FlowFile content into heap on a fairly heap-constrained JVM you can cause OOME, etc. that effects all flows.
> I believe writing to the Content Repository directly is the correct default. However, there are cases where keeping the FlowFile content in memory and on-heap still makes sense even for a Stateless Process Group. Two cases in particular:
>  # The flow is designed such that it only ever keeps small a small amount of data in-flight at a given time. If it never will buffer more than 5-10 MB of data it's still quite safe.
>  # Environments where the only thing running is a Stateless Group. With the prevalence of running NiFi on Kubernetes now, we see many users create a lot of tiny NiFi instances instead of a few huge instances. In this paradigm, it's far more common to have a flow that only runs a Stateless Group or where the Stateless Group is the majority of the processing logic.
> In order to accommodate these cases, we should allow buffering the FlowFile content in memory/heap.
> h1. Proposal
> My proposal, then, is to allow a Process Group, when its Execution Engine is configured for Stateless, to also be able to configure whether FlowFile content is written to the Content Repository or buffered in memory.
> This would result in a new configuration element in the UI, and the default will write to the Content Repository, but it would become configurable.
> An important caveat is that we should NOT allow it to be configured differently from one Process Group to its child. This is similar to how we forbid Group A from having an Execution Engine of Stateless while its child group has an Execution Engine of Standard.
> h1. API Changes
> This will require changes to typical REST endpoint / flow configuration objects: ProcessGroupDTO, VersionedProcessGroup, ProcessGroup, endpoint mergers, etc. in order to provide the plumbing.
> h1. Framework Changes
> This will also require updates to the framework for this to be implemented, but the Content Repository implementation already exists. It may require moving it into a different maven module that can be more readily shared, but I don't expect this to require significant machinery otherwise.
> There will also be necessary updates to the flow inheritance modules to ensure that we apply updates appropriately.
> h1. Documentation Changes
> We will need to update the user guide to explain this new capability and the tradeoffs of in-memory vs. Content Repository.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)