You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Peter Wicks (pwicks)" <pw...@micron.com> on 2018/08/10 21:39:10 UTC
After 1.7.1 upgrade, no Provenance data is visible
After upgrading our NiFi instances to 1.7.1 we are not able to see Provenance data anymore in the UI. We see this across about a dozen instances.
In the UI it tells me provenance is available for about the last 24 hours, and I can see that files have moved in and out of the processor in the last 5 min. In the logs, I can see it query provenance, and that the query returns 0 results.
Thoughts? I saw a few tickets related to Provenance in 1.7, but not sure if they have an impact.
Here are our properties:
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/data/nifi/repositories/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.warm.cache.frequency=1 hour
Thanks,
Peter
RE: [EXT] Re: After 1.7.1 upgrade, no Provenance data is visible
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Thanks Mike, that fixed it.
--Peter
From: Michael Moser [mailto:moser.mw@gmail.com]
Sent: Friday, August 10, 2018 3:49 PM
To: users@nifi.apache.org
Subject: [EXT] Re: After 1.7.1 upgrade, no Provenance data is visible
Hi Peter,
There was a change to provenance related access policies in 1.7.0. Check out the Migration Guide [1] for 1.7.0. It talks about what you'll need to do.
[1] - https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
-- Mike
On Fri, Aug 10, 2018 at 5:39 PM Peter Wicks (pwicks) <pw...@micron.com>> wrote:
After upgrading our NiFi instances to 1.7.1 we are not able to see Provenance data anymore in the UI. We see this across about a dozen instances.
In the UI it tells me provenance is available for about the last 24 hours, and I can see that files have moved in and out of the processor in the last 5 min. In the logs, I can see it query provenance, and that the query returns 0 results.
Thoughts? I saw a few tickets related to Provenance in 1.7, but not sure if they have an impact.
Here are our properties:
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id<http://nifi.provenance.repository.encryption.key.id>=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/data/nifi/repositories/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.warm.cache.frequency=1 hour
Thanks,
Peter
Re: After 1.7.1 upgrade, no Provenance data is visible
Posted by Michael Moser <mo...@gmail.com>.
Hi Peter,
There was a change to provenance related access policies in 1.7.0. Check
out the Migration Guide [1] for 1.7.0. It talks about what you'll need to
do.
[1] - https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
-- Mike
On Fri, Aug 10, 2018 at 5:39 PM Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> After upgrading our NiFi instances to 1.7.1 we are not able to see
> Provenance data anymore in the UI. We see this across about a dozen
> instances.
>
> In the UI it tells me provenance is available for about the last 24 hours,
> and I can see that files have moved in and out of the processor in the last
> 5 min. In the logs, I can see it query provenance, and that the query
> returns 0 results.
>
>
>
> Thoughts? I saw a few tickets related to Provenance in 1.7, but not sure
> if they have an impact.
>
>
>
> Here are our properties:
>
>
>
> # Provenance Repository Properties
>
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
>
> nifi.provenance.repository.debug.frequency=1_000_000
>
> nifi.provenance.repository.encryption.key.provider.implementation=
>
> nifi.provenance.repository.encryption.key.provider.location=
>
> nifi.provenance.repository.encryption.key.id=
>
> nifi.provenance.repository.encryption.key=
>
>
>
> # Persistent Provenance Repository Properties
>
>
> nifi.provenance.repository.directory.default=/data/nifi/repositories/provenance_repository
>
> nifi.provenance.repository.max.storage.time=24 hours
>
> nifi.provenance.repository.max.storage.size=1 GB
>
> nifi.provenance.repository.rollover.time=30 secs
>
> nifi.provenance.repository.rollover.size=100 MB
>
> nifi.provenance.repository.query.threads=2
>
> nifi.provenance.repository.index.threads=2
>
> nifi.provenance.repository.compress.on.rollover=true
>
> nifi.provenance.repository.always.sync=false
>
> nifi.provenance.repository.journal.count=16
>
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
>
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details
>
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> Filename, ProcessorID, Relationship
>
> # FlowFile Attributes that should be indexed and made searchable. Some
> examples to consider are filename, uuid, mime.type
>
> nifi.provenance.repository.indexed.attributes=
>
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
>
> # but should provide better performance
>
> nifi.provenance.repository.index.shard.size=500 MB
>
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
>
> # the repository. If the length of any attribute exceeds this value, it
> will be truncated when the event is retrieved.
>
> nifi.provenance.repository.max.attribute.length=65536
>
> nifi.provenance.repository.concurrent.merge.threads=2
>
> nifi.provenance.repository.warm.cache.frequency=1 hour
>
>
>
> Thanks,
>
> Peter
>