You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Nguyen, Kyle" <Ky...@mlp.com.INVALID> on 2025/03/04 15:53:44 UTC
Minimum provenance to keep nifi property
Hi. Is there a “minimum provenance” to keep setting for a given processor? Use-case is
1. User asks us to check provenance on one of their “weekly” workflows (for data errors, etc.)
2. The “weekly” task’s provenance got deleted—perhaps from other, more frequently executed processors, due to our quota/limit settings.
3. So it would be nice if there was a “minimum provenance to keep” setting.
We’re aware of and currently have these properties in `nifi.properties`.
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
[cid:image001.png@01DB8CF3.0A1EBDD0]
Kyle Nguyen
Corporate Technology, Software Engineer
Millennium Management LLC
399 Park Avenue | New York, NY 10022
📞 +1.212.708.1366 | 📱 +1.929.837.1788
mlp.com<https://www.mlp.com/home/>
######################################################################
The information contained in this communication is confidential and
may contain information that is privileged or exempt from disclosure
under applicable law. If you are not a named addressee, please notify
the sender immediately and delete this email from your system.
If you have received this communication, and are not a named
recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
######################################################################
Re: Minimum provenance to keep nifi property
Posted by Mark Payne <ma...@hotmail.com>.
Hi Kyle,
There’s no minimum amount to keep - it will always keep everything that it can keep, up to the configured max. So in your case once you hit 10 GB of provenance events it will age off the older events to make room for newer events. And anything over 30 days will also be eliminated, for compliance types of reasons. Note that the provenance repository only stores information related to lineage and FlowFile attributes. It does not store FlowFile content. The content is stored in the Content Repository and the amount of data to keep available for viewing through provenance is controlled by the “nifi.content.repository.archive.max.retention.period” and “nifi.content.repository.archive.max.usage.percentage" properties.
Thanks
-Mark
On Mar 4, 2025, at 10:53 AM, Nguyen, Kyle <Ky...@mlp.com.INVALID> wrote:
Hi. Is there a “minimum provenance” to keep setting for a given processor? Use-case is
1. User asks us to check provenance on one of their “weekly” workflows (for data errors, etc.)
2. The “weekly” task’s provenance got deleted—perhaps from other, more frequently executed processors, due to our quota/limit settings.
3. So it would be nice if there was a “minimum provenance to keep” setting.
We’re aware of and currently have these properties in `nifi.properties`.
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
[cid:image001.png@01DB8CF3.0A1EBDD0]
Kyle Nguyen
Corporate Technology, Software Engineer
Millennium Management LLC
399 Park Avenue | New York, NY 10022
📞 +1.212.708.1366 | 📱 +1.929.837.1788
mlp.com<https://www.mlp.com/home/>
######################################################################
The information contained in this communication is confidential and
may contain information that is privileged or exempt from disclosure
under applicable law. If you are not a named addressee, please notify
the sender immediately and delete this email from your system.
If you have received this communication, and are not a named
recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
######################################################################