[jira] [Created] (NIP-40) Content and FlowFile Repository API Cleanup

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIP-40:
-----------------------------

             Summary: Content and FlowFile Repository API Cleanup
                 Key: NIP-40
                 URL: https://issues.apache.org/jira/browse/NIP-40
             Project: NiFi Improvement Proposal
          Issue Type: Improvement
            Reporter: Mark Payne


h2. Motivation

Several small, related gaps in the Content Repository and FlowFile Repository
API (nifi-framework-api) make it harder than necessary to implement t he repository interfaces and to reason about repository behavior. This NIP
groups a few backward-compatible cleanups to that API.
 # *No component or Connector context on writes.* When a ProcessSession creates
content or commits FlowFile changes, it calls {{ContentRepository.create(...)}}
and {{{}FlowFileRepository.updateRepository(...){}}}. Neither call tells the
repository which component the session is acting on behalf of, nor which
Connector (managed flow) that component belongs to, so the repository cannot
attribute an operation to a component or organize data by Connector. There have been several instances when I wanted to know which component a Content Claim was for; e.g., if we know that it's for a Processor that statistically will create large FlowFiles we may not want to append its content to an existing Resource Claim but instead create a new Resource Claim. This would help avoid scenarios where Content Repository holds onto a large file just because one small FlowFile shares the same Resource Claim.
 # *The session depends on framework-internal stream classes.*
{{StandardProcessSession}} reads content through {{{}ContentClaimInputStream{}}}, and the write cache reuses claims through {{{}ContentClaimOutputStream.newContentClaim(){}}}.
These abstractions are good and make sense, but both classes live in framework internals (nifi-framework-components), not in the
more public {{{}nifi-framework-api{}}}, so a repository that wants to provide its own read stream or claim-reuse behavior has no public abstraction to implement and must bind to framework internals.
 # *Content Repository should own the maximum appendable claim size.* The claim rollover size is read from the {{nifi.content.claim.max.appendable.size}} property and threaded to the write cache, while a repository that has its own notion of where a storage unit ends has no way to declare it. The two values must be kept
consistent by hand. While participation from Process Session is necessary for the max appendable size to work properly, it should be the responsibility of the pluggable {{ContentRepository}} to dictate what max size should be used.

h2. Proposed Solution

Three backward-compatible additions to the repository SPI:

 * *Component/Connector context.* Pass a small, immutable context object on each
create/update describing the component and Connector the session is acting for.
It is populated by the framework and passed straight through; stock repositories
ignore it.
 * *Public stream abstractions.* Expose the content-claim input-stream
abstraction and the output-stream claim-reuse contract in nifi-framework-api,
and have {{StandardProcessSession}} and the write cache depend on those public
types rather than the framework-internal classes.
 * *Repository-owned appendable size.* Let {{ContentRepository}} report its
maximum appendable claim size, and have the framework source the rollover size
from the repository. The existing property remains the default the stock
repository reports.
All additions are default methods or newly exposed types; no existing type or
method is removed, changed, or deprecated. The stock FileSystemRepository and
WriteAheadFlowFileRepository behave exactly as they do today.
h2. Framework API Changes (nifi-framework-api)

*Component/Connector context*

 * New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}}, {{{}LOSS_INTOLERANT{}}}). While we could keep with the existing {{boolean lossTolerant}} the general pattern that NiFi has tended toward is to drop booleans in interfaces/APIs in favor of more descriptive enums.
 * New {{ContentClaimCreationContext}} (component id, Connector id, loss
tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
 * New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}} and
{{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>, FlowFileUpdateContext){}}},
each delegating to the existing method.{*}{*}

 

*Public stream abstractions*
 * Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from {{nifi-framework-components}} to {{nifi-framework-api}}

*Repository-owned appendable size*
 * New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}} returning
the configured default (so stock behavior is unchanged); the framework sources
the rollover size from the repository. The value must be a cheap in-memory read,
since it is consulted on the write hot path.
The existing {{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and the
{{nifi.content.claim.max.appendable.size}} property are all retained and not
deprecated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)