Deadlock bug in PersistentProvenanceRepository

[Brian Davis <br...@gmail.com>]

I have been running nifi for about a week on a machine that has 32 cpus.
During this week I have seen the nifi instance lockup about 5 times (it
could of been more if it was not a weekend).  Lockup being no flowfiles
moving.  All the threads are in the PersistentProvenanceRepository.  I have
submitted a bug here:

https://issues.apache.org/jira/browse/NIFI-2395

I think the fix is as easy as changing a || to a &&.  But there are a lot
of parts to this class and I could easily be missing something.  I tried to
document where all my threads were.  I believe this bug would hit any
system that runs nifi that has low to unused cpus.  I know it effects 0.6.0
and it looks like it is in 0.7.0.  It is a very crippling bug.  I do not
think I have lost any data, but cannot be sure.

If you have seen flows that just stop processing data, this could be the
cause.

You have a good one.

Brian

Re: Deadlock bug in PersistentProvenanceRepository

[Brian Davis <br...@gmail.com>]

I will get you all of the stuff you asked for tomorrow morning.

Thank you for making the code easy to follow and understand.  Plus, taking
your time on this bug that is affecting me.

You have a good one.

Brian



On Tue, Jul 26, 2016 at 9:50 PM, Joe Witt <jo...@gmail.com> wrote:

> Thanks Brian.  Will discuss on the JIRA.  If you could possibly share
> the stack dumps for anything involving the prov repo that would be
> really helpful.
>
> Thanks for reporting and also providing really helpful context/analysis.
>
> Joe
>
> On Tue, Jul 26, 2016 at 9:23 PM, Brian Davis
> <br...@gmail.com> wrote:
> > I have been running nifi for about a week on a machine that has 32 cpus.
> > During this week I have seen the nifi instance lockup about 5 times (it
> > could of been more if it was not a weekend).  Lockup being no flowfiles
> > moving.  All the threads are in the PersistentProvenanceRepository.  I
> have
> > submitted a bug here:
> >
> > https://issues.apache.org/jira/browse/NIFI-2395
> >
> > I think the fix is as easy as changing a || to a &&.  But there are a
> lot of
> > parts to this class and I could easily be missing something.  I tried to
> > document where all my threads were.  I believe this bug would hit any
> system
> > that runs nifi that has low to unused cpus.  I know it effects 0.6.0 and
> it
> > looks like it is in 0.7.0.  It is a very crippling bug.  I do not think I
> > have lost any data, but cannot be sure.
> >
> > If you have seen flows that just stop processing data, this could be the
> > cause.
> >
> > You have a good one.
> >
> > Brian
>

Re: Deadlock bug in PersistentProvenanceRepository

[Joe Witt <jo...@gmail.com>]

Thanks Brian.  Will discuss on the JIRA.  If you could possibly share
the stack dumps for anything involving the prov repo that would be
really helpful.

Thanks for reporting and also providing really helpful context/analysis.

Joe

On Tue, Jul 26, 2016 at 9:23 PM, Brian Davis
<br...@gmail.com> wrote:
> I have been running nifi for about a week on a machine that has 32 cpus.
> During this week I have seen the nifi instance lockup about 5 times (it
> could of been more if it was not a weekend).  Lockup being no flowfiles
> moving.  All the threads are in the PersistentProvenanceRepository.  I have
> submitted a bug here:
>
> https://issues.apache.org/jira/browse/NIFI-2395
>
> I think the fix is as easy as changing a || to a &&.  But there are a lot of
> parts to this class and I could easily be missing something.  I tried to
> document where all my threads were.  I believe this bug would hit any system
> that runs nifi that has low to unused cpus.  I know it effects 0.6.0 and it
> looks like it is in 0.7.0.  It is a very crippling bug.  I do not think I
> have lost any data, but cannot be sure.
>
> If you have seen flows that just stop processing data, this could be the
> cause.
>
> You have a good one.
>
> Brian