You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by dan young <da...@gmail.com> on 2018/01/17 00:26:35 UTC
unable to search provenance repository
Hello,
We're running a secure 3 node 1.4 cluster, and for some reason we're not
able to search processor provenance event. i.e. trying to search a for a
particular event by filename, but the results always come back empty, even
though when I view the provenance event history I can see the event. Has
anyone seen this, or can offer any suggestions on how to debug why I'm not
able to search?
I've tried shutting the cluster down and deleting all the repositories a
couple of times....
Here's my relevant config:
nifi.provenance.repository.implementation=
org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=
/opt/nifi-common/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=5 GB
nifi.provenance.repository.rollover.time=5 min
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
Filename, ProcessorID, Relationship
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=500 MB
Re: unable to search provenance repository
Posted by dan young <da...@gmail.com>.
Joe,
looked at dmesg and see a few of these on one of the nodes...I wonder if
it's an IO issue. I'll do some digging around
[5492401.012087] INFO: task java:15838 blocked for more than 120 seconds.
....
....
....
On Wed, Jan 17, 2018 at 8:43 AM Joe Witt <jo...@gmail.com> wrote:
> Well, I think i'm hoping it was the restart and not the change of
> shard size because one I can understand and the other I cannot :)
>
> If you think you run into this again can you please get thread dumps
> and share them?
>
> Thanks!
>
> On Wed, Jan 17, 2018 at 10:41 AM, dan young <da...@gmail.com> wrote:
> > Heya Joe,
> >
> > I'm not sure if this had anything to do with it, but I increased the
> > nifi.provenance.repository.index.shard.size to 4GB from the default
> 500MB,
> > did a rolling restart of the nodes, and now the search seems to be
> working.
> > Not sure if it was that or the restart....but I'll keep an eye on it.
> >
> > Regards,
> >
> > Dano
> >
> >
> > On Tue, Jan 16, 2018 at 7:57 PM Joe Witt <jo...@gmail.com> wrote:
> >>
> >> dan - i've seen behavior that frustrated me like this at times but it
> >> was almost always based on me not realizing the timezone settings or
> >> something else related to my query time versus the time on the system.
> >> I believe you can activate more detailed logging for that class and
> >> others to see information about the queries themselves. I'll try some
> >> of this tomorrow and share more findings if able.
> >>
> >> If no luck then please be sure to file a JIRA.
> >>
> >> Thanks
> >>
> >> On Tue, Jan 16, 2018 at 7:26 PM, dan young <da...@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > We're running a secure 3 node 1.4 cluster, and for some reason we're
> not
> >> > able to search processor provenance event. i.e. trying to search a
> for
> >> > a
> >> > particular event by filename, but the results always come back empty,
> >> > even
> >> > though when I view the provenance event history I can see the event.
> >> > Has
> >> > anyone seen this, or can offer any suggestions on how to debug why I'm
> >> > not
> >> > able to search?
> >> >
> >> > I've tried shutting the cluster down and deleting all the
> repositories a
> >> > couple of times....
> >> >
> >> > Here's my relevant config:
> >> >
> >> >
> >> >
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> >> >
> >> > nifi.provenance.repository.debug.frequency=1_000_000
> >> >
> >> > nifi.provenance.repository.encryption.key.provider.implementation=
> >> >
> >> > nifi.provenance.repository.encryption.key.provider.location=
> >> >
> >> > nifi.provenance.repository.encryption.key.id=
> >> >
> >> > nifi.provenance.repository.encryption.key=
> >> >
> >> >
> >> > # Persistent Provenance Repository Properties
> >> >
> >> >
> >> >
> nifi.provenance.repository.directory.default=/opt/nifi-common/provenance_repository
> >> >
> >> > nifi.provenance.repository.max.storage.time=24 hours
> >> >
> >> > nifi.provenance.repository.max.storage.size=5 GB
> >> >
> >> > nifi.provenance.repository.rollover.time=5 min
> >> >
> >> > nifi.provenance.repository.rollover.size=100 MB
> >> >
> >> > nifi.provenance.repository.query.threads=2
> >> >
> >> > nifi.provenance.repository.index.threads=2
> >> >
> >> > nifi.provenance.repository.compress.on.rollover=true
> >> >
> >> > nifi.provenance.repository.always.sync=false
> >> >
> >> > nifi.provenance.repository.journal.count=16
> >> >
> >> > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> >> > Filename,
> >> > ProcessorID, Relationship
> >> >
> >> > nifi.provenance.repository.indexed.attributes=
> >> >
> >> > nifi.provenance.repository.index.shard.size=500 MB
>
Re: unable to search provenance repository
Posted by Joe Witt <jo...@gmail.com>.
Well, I think i'm hoping it was the restart and not the change of
shard size because one I can understand and the other I cannot :)
If you think you run into this again can you please get thread dumps
and share them?
Thanks!
On Wed, Jan 17, 2018 at 10:41 AM, dan young <da...@gmail.com> wrote:
> Heya Joe,
>
> I'm not sure if this had anything to do with it, but I increased the
> nifi.provenance.repository.index.shard.size to 4GB from the default 500MB,
> did a rolling restart of the nodes, and now the search seems to be working.
> Not sure if it was that or the restart....but I'll keep an eye on it.
>
> Regards,
>
> Dano
>
>
> On Tue, Jan 16, 2018 at 7:57 PM Joe Witt <jo...@gmail.com> wrote:
>>
>> dan - i've seen behavior that frustrated me like this at times but it
>> was almost always based on me not realizing the timezone settings or
>> something else related to my query time versus the time on the system.
>> I believe you can activate more detailed logging for that class and
>> others to see information about the queries themselves. I'll try some
>> of this tomorrow and share more findings if able.
>>
>> If no luck then please be sure to file a JIRA.
>>
>> Thanks
>>
>> On Tue, Jan 16, 2018 at 7:26 PM, dan young <da...@gmail.com> wrote:
>> > Hello,
>> >
>> > We're running a secure 3 node 1.4 cluster, and for some reason we're not
>> > able to search processor provenance event. i.e. trying to search a for
>> > a
>> > particular event by filename, but the results always come back empty,
>> > even
>> > though when I view the provenance event history I can see the event.
>> > Has
>> > anyone seen this, or can offer any suggestions on how to debug why I'm
>> > not
>> > able to search?
>> >
>> > I've tried shutting the cluster down and deleting all the repositories a
>> > couple of times....
>> >
>> > Here's my relevant config:
>> >
>> >
>> > nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
>> >
>> > nifi.provenance.repository.debug.frequency=1_000_000
>> >
>> > nifi.provenance.repository.encryption.key.provider.implementation=
>> >
>> > nifi.provenance.repository.encryption.key.provider.location=
>> >
>> > nifi.provenance.repository.encryption.key.id=
>> >
>> > nifi.provenance.repository.encryption.key=
>> >
>> >
>> > # Persistent Provenance Repository Properties
>> >
>> >
>> > nifi.provenance.repository.directory.default=/opt/nifi-common/provenance_repository
>> >
>> > nifi.provenance.repository.max.storage.time=24 hours
>> >
>> > nifi.provenance.repository.max.storage.size=5 GB
>> >
>> > nifi.provenance.repository.rollover.time=5 min
>> >
>> > nifi.provenance.repository.rollover.size=100 MB
>> >
>> > nifi.provenance.repository.query.threads=2
>> >
>> > nifi.provenance.repository.index.threads=2
>> >
>> > nifi.provenance.repository.compress.on.rollover=true
>> >
>> > nifi.provenance.repository.always.sync=false
>> >
>> > nifi.provenance.repository.journal.count=16
>> >
>> > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
>> > Filename,
>> > ProcessorID, Relationship
>> >
>> > nifi.provenance.repository.indexed.attributes=
>> >
>> > nifi.provenance.repository.index.shard.size=500 MB
Re: unable to search provenance repository
Posted by dan young <da...@gmail.com>.
Heya Joe,
I'm not sure if this had anything to do with it, but I increased
the nifi.provenance.repository.index.shard.size to 4GB from the default
500MB, did a rolling restart of the nodes, and now the search seems to be
working. Not sure if it was that or the restart....but I'll keep an eye on
it.
Regards,
Dano
On Tue, Jan 16, 2018 at 7:57 PM Joe Witt <jo...@gmail.com> wrote:
> dan - i've seen behavior that frustrated me like this at times but it
> was almost always based on me not realizing the timezone settings or
> something else related to my query time versus the time on the system.
> I believe you can activate more detailed logging for that class and
> others to see information about the queries themselves. I'll try some
> of this tomorrow and share more findings if able.
>
> If no luck then please be sure to file a JIRA.
>
> Thanks
>
> On Tue, Jan 16, 2018 at 7:26 PM, dan young <da...@gmail.com> wrote:
> > Hello,
> >
> > We're running a secure 3 node 1.4 cluster, and for some reason we're not
> > able to search processor provenance event. i.e. trying to search a for a
> > particular event by filename, but the results always come back empty,
> even
> > though when I view the provenance event history I can see the event. Has
> > anyone seen this, or can offer any suggestions on how to debug why I'm
> not
> > able to search?
> >
> > I've tried shutting the cluster down and deleting all the repositories a
> > couple of times....
> >
> > Here's my relevant config:
> >
> >
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> >
> > nifi.provenance.repository.debug.frequency=1_000_000
> >
> > nifi.provenance.repository.encryption.key.provider.implementation=
> >
> > nifi.provenance.repository.encryption.key.provider.location=
> >
> > nifi.provenance.repository.encryption.key.id=
> >
> > nifi.provenance.repository.encryption.key=
> >
> >
> > # Persistent Provenance Repository Properties
> >
> >
> nifi.provenance.repository.directory.default=/opt/nifi-common/provenance_repository
> >
> > nifi.provenance.repository.max.storage.time=24 hours
> >
> > nifi.provenance.repository.max.storage.size=5 GB
> >
> > nifi.provenance.repository.rollover.time=5 min
> >
> > nifi.provenance.repository.rollover.size=100 MB
> >
> > nifi.provenance.repository.query.threads=2
> >
> > nifi.provenance.repository.index.threads=2
> >
> > nifi.provenance.repository.compress.on.rollover=true
> >
> > nifi.provenance.repository.always.sync=false
> >
> > nifi.provenance.repository.journal.count=16
> >
> > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> Filename,
> > ProcessorID, Relationship
> >
> > nifi.provenance.repository.indexed.attributes=
> >
> > nifi.provenance.repository.index.shard.size=500 MB
>
Re: unable to search provenance repository
Posted by Joe Witt <jo...@gmail.com>.
dan - i've seen behavior that frustrated me like this at times but it
was almost always based on me not realizing the timezone settings or
something else related to my query time versus the time on the system.
I believe you can activate more detailed logging for that class and
others to see information about the queries themselves. I'll try some
of this tomorrow and share more findings if able.
If no luck then please be sure to file a JIRA.
Thanks
On Tue, Jan 16, 2018 at 7:26 PM, dan young <da...@gmail.com> wrote:
> Hello,
>
> We're running a secure 3 node 1.4 cluster, and for some reason we're not
> able to search processor provenance event. i.e. trying to search a for a
> particular event by filename, but the results always come back empty, even
> though when I view the provenance event history I can see the event. Has
> anyone seen this, or can offer any suggestions on how to debug why I'm not
> able to search?
>
> I've tried shutting the cluster down and deleting all the repositories a
> couple of times....
>
> Here's my relevant config:
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
>
> nifi.provenance.repository.debug.frequency=1_000_000
>
> nifi.provenance.repository.encryption.key.provider.implementation=
>
> nifi.provenance.repository.encryption.key.provider.location=
>
> nifi.provenance.repository.encryption.key.id=
>
> nifi.provenance.repository.encryption.key=
>
>
> # Persistent Provenance Repository Properties
>
> nifi.provenance.repository.directory.default=/opt/nifi-common/provenance_repository
>
> nifi.provenance.repository.max.storage.time=24 hours
>
> nifi.provenance.repository.max.storage.size=5 GB
>
> nifi.provenance.repository.rollover.time=5 min
>
> nifi.provenance.repository.rollover.size=100 MB
>
> nifi.provenance.repository.query.threads=2
>
> nifi.provenance.repository.index.threads=2
>
> nifi.provenance.repository.compress.on.rollover=true
>
> nifi.provenance.repository.always.sync=false
>
> nifi.provenance.repository.journal.count=16
>
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID, Relationship
>
> nifi.provenance.repository.indexed.attributes=
>
> nifi.provenance.repository.index.shard.size=500 MB