[DISCUSS] NIP-38: Pluggable Flow Persistence and Audit Storage

[Mark Payne <ma...@hotmail.com>]

Team,

I have drafted NIP-38 [1] proposing that we make pluggable the two parts of the framework that are hardcoded to write local files: the dataflow (flow.json.gz) and the flow-change history (the audit store under nifi.database.directory).

For the dataflow, this adds a FlowPersistenceProvider extension point; VersionedDataflow moves into nifi-framework-api so the provider API can be typed, and saveFlow takes a FlowPersistenceContext so we can later add things like a change description without breaking providers.

For the audit history, the existing AuditService interface moves into nifi-framework-api and becomes selectable, defaulting to today's EntityStoreAuditService.

The intent is for the default behavior to remain unchanged. However, this would provide appropriate abstractions for concepts that are now hard-coded, with rather minimal surface area / updates.

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIP-38

Re: [DISCUSS] NIP-38: Pluggable Flow Persistence and Audit Storage

[Kevin Doran <kd...@apache.org>]

This seems like a natural framework extension point. I am supportive
of the concept.

Cheers,
Kevin

On Tue, Aug 11, 2026 at 3:54 PM Mark Payne <ma...@hotmail.com> wrote:
>
> Team,
>
> I have drafted NIP-38 [1] proposing that we make pluggable the two parts of the framework that are hardcoded to write local files: the dataflow (flow.json.gz) and the flow-change history (the audit store under nifi.database.directory).
>
> For the dataflow, this adds a FlowPersistenceProvider extension point; VersionedDataflow moves into nifi-framework-api so the provider API can be typed, and saveFlow takes a FlowPersistenceContext so we can later add things like a change description without breaking providers.
>
> For the audit history, the existing AuditService interface moves into nifi-framework-api and becomes selectable, defaulting to today's EntityStoreAuditService.
>
> The intent is for the default behavior to remain unchanged. However, this would provide appropriate abstractions for concepts that are now hard-coded, with rather minimal surface area / updates.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIP-38

Re: [DISCUSS] NIP-38: Pluggable Flow Persistence and Audit Storage

[David Handermann <ex...@apache.org>]

Mark,

Thanks for proposing these new framework extensions, I support the
approach and configuration options described in the proposal.

The only question that comes to mind is regarding the naming of the
AuditService. Although it has remained unchanged for years, this seems
like a good opportunity to adjust the naming. NIP-7 [1] introduced the
FlowActionReporter as a method to export audit information, using the
Action naming convention. The FlowActionReporter currently plugs into
the EntityStoreAuditService, but the wiring could be adjusted to call
FlowActionReporter directly under this new proposal. So when it comes
to naming, following the convention of other Repositories, what do you
think about calling this FlowActionRepository? For something more
verbose, FlowActionAuditRepository could also work.

Regards,
David Handermann

[1] https://issues.apache.org/jira/browse/NIP-7

On Wed, Aug 12, 2026 at 5:56 AM Matt Burgess <ma...@apache.org> wrote:
>
> +1 from me!
>
> On Tue, Aug 11, 2026 at 3:54 PM Mark Payne <ma...@hotmail.com> wrote:
>
> > Team,
> >
> > I have drafted NIP-38 [1] proposing that we make pluggable the two parts
> > of the framework that are hardcoded to write local files: the dataflow
> > (flow.json.gz) and the flow-change history (the audit store under
> > nifi.database.directory).
> >
> > For the dataflow, this adds a FlowPersistenceProvider extension point;
> > VersionedDataflow moves into nifi-framework-api so the provider API can be
> > typed, and saveFlow takes a FlowPersistenceContext so we can later add
> > things like a change description without breaking providers.
> >
> > For the audit history, the existing AuditService interface moves into
> > nifi-framework-api and becomes selectable, defaulting to today's
> > EntityStoreAuditService.
> >
> > The intent is for the default behavior to remain unchanged. However, this
> > would provide appropriate abstractions for concepts that are now
> > hard-coded, with rather minimal surface area / updates.
> >
> > Thanks
> > -Mark
> >
> > [1] https://issues.apache.org/jira/browse/NIP-38
> >

Re: [DISCUSS] NIP-38: Pluggable Flow Persistence and Audit Storage

[Matt Burgess <ma...@apache.org>]

+1 from me!

On Tue, Aug 11, 2026 at 3:54 PM Mark Payne <ma...@hotmail.com> wrote:

> Team,
>
> I have drafted NIP-38 [1] proposing that we make pluggable the two parts
> of the framework that are hardcoded to write local files: the dataflow
> (flow.json.gz) and the flow-change history (the audit store under
> nifi.database.directory).
>
> For the dataflow, this adds a FlowPersistenceProvider extension point;
> VersionedDataflow moves into nifi-framework-api so the provider API can be
> typed, and saveFlow takes a FlowPersistenceContext so we can later add
> things like a change description without breaking providers.
>
> For the audit history, the existing AuditService interface moves into
> nifi-framework-api and becomes selectable, defaulting to today's
> EntityStoreAuditService.
>
> The intent is for the default behavior to remain unchanged. However, this
> would provide appropriate abstractions for concepts that are now
> hard-coded, with rather minimal surface area / updates.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIP-38
>