You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Daniel Watson <dc...@gmail.com> on 2018/08/17 14:07:45 UTC
NiFi hangs at Computing Data Lineage... 100% for a specific flow
Anyone have any issues with the data lineage screen? My NiFi instance
can't compute the data lineage for a specific flow. It worked originally,
then after running some files through, it no longer will. It gets stuck at
100% and the UI shows nothing.
The flow is...
list files -> update att -> order -> fetch files -> update att -> put files
I removed the provenance directory and restarted, and it did the same thing.
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Daniel Watson <dc...@gmail.com>.
Mark,
I just read somewhere that the EnforceOrder processor is not meant to be
used when numbers will be "skipped". Since im using a timestamp it wont
work for me. So I'll need to find another way to do that ordering. I looked
at the PriorityAttributePrioritizer, but that doesn't seem to be able to
queue for some amount of time before proceeding. Is that correct?
Thanks,
Dan
On Fri, Aug 17, 2018 at 11:13 AM Daniel Watson <dc...@gmail.com> wrote:
> Mark,
>
> Thanks! That worked in allowing the DP page to load. Now it exposes the
> underlying problem, which is that the EnforceOrder processor seems to be
> adding hundreds of "attributes modified" events, even when no attributes
> are being modified. Does this have something to do with the flow files in
> the "wait" state cycling through the processor over and over again? In the
> examples of that processor I see people looping the "wait" event back
> around to the processor so that the files are held in that processor until
> the batch is complete, so that's what I did. Is that an intended usage of
> it? If so, is there something that can be done about those DP events?
>
> Thanks,
> Dan
>
> On Fri, Aug 17, 2018 at 10:30 AM Mark Payne <ma...@hotmail.com> wrote:
>
>> OK, my recommendation would be to change to the
>> WriteAheadProvenanceRepository. You can do this by
>> changing the value of the "nifi.provenance.repository.implementation"
>> property in nifi.properties to
>> "org.apache.nifi.provenance.WriteAheadProvenanceRepository"
>>
>> The WriteAhead implementation was written several releases back, but we
>> left Persistent Provenance Repo
>> as the default just to ensure that the Write-Ahead implementation was
>> stable enough. At this point, it's pretty
>> clear that it is not only much faster than the Persistent Prov Repo but
>> also far more stable. There is a JIRA [1]
>> to make this the new default.
>>
>> The nice thing is that it is backward compatible, so if you change to the
>> WriteAhead impl and restart, you should
>> still be able to query all of your existing data without losing anything.
>>
>> Thanks
>> -Mark
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-5482
>>
>>
>> On Aug 17, 2018, at 10:26 AM, Daniel Watson <dc...@gmail.com> wrote:
>>
>> Yes.
>>
>> On Fri, Aug 17, 2018 at 10:19 AM Mark Payne <ma...@hotmail.com> wrote:
>>
>>> OK, thanks. Are you using the default implementation of the Provenance
>>> Repository? I.e., the PersistentProvenanceRepository?
>>>
>>>
>>> On Aug 17, 2018, at 10:10 AM, Daniel Watson <dc...@gmail.com>
>>> wrote:
>>>
>>> Mark,
>>>
>>> 1.7.0
>>> 06/19/2018 21:55:30 EDT
>>> Tagged nifi-1.7.0-RC1
>>>
>>> Thanks
>>>
>>> On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com>
>>> wrote:
>>>
>>>> Hi Daniel,
>>>>
>>>> What version of NiFi are you running?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> > On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Anyone have any issues with the data lineage screen? My NiFi
>>>> instance can't compute the data lineage for a specific flow. It worked
>>>> originally, then after running some files through, it no longer will. It
>>>> gets stuck at 100% and the UI shows nothing.
>>>> >
>>>> > The flow is...
>>>> >
>>>> > list files -> update att -> order -> fetch files -> update att -> put
>>>> files
>>>> >
>>>> > I removed the provenance directory and restarted, and it did the same
>>>> thing.
>>>>
>>>>
>>>
>>
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Daniel Watson <dc...@gmail.com>.
Mark,
Thanks! That worked in allowing the DP page to load. Now it exposes the
underlying problem, which is that the EnforceOrder processor seems to be
adding hundreds of "attributes modified" events, even when no attributes
are being modified. Does this have something to do with the flow files in
the "wait" state cycling through the processor over and over again? In the
examples of that processor I see people looping the "wait" event back
around to the processor so that the files are held in that processor until
the batch is complete, so that's what I did. Is that an intended usage of
it? If so, is there something that can be done about those DP events?
Thanks,
Dan
On Fri, Aug 17, 2018 at 10:30 AM Mark Payne <ma...@hotmail.com> wrote:
> OK, my recommendation would be to change to the
> WriteAheadProvenanceRepository. You can do this by
> changing the value of the "nifi.provenance.repository.implementation"
> property in nifi.properties to
> "org.apache.nifi.provenance.WriteAheadProvenanceRepository"
>
> The WriteAhead implementation was written several releases back, but we
> left Persistent Provenance Repo
> as the default just to ensure that the Write-Ahead implementation was
> stable enough. At this point, it's pretty
> clear that it is not only much faster than the Persistent Prov Repo but
> also far more stable. There is a JIRA [1]
> to make this the new default.
>
> The nice thing is that it is backward compatible, so if you change to the
> WriteAhead impl and restart, you should
> still be able to query all of your existing data without losing anything.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-5482
>
>
> On Aug 17, 2018, at 10:26 AM, Daniel Watson <dc...@gmail.com> wrote:
>
> Yes.
>
> On Fri, Aug 17, 2018 at 10:19 AM Mark Payne <ma...@hotmail.com> wrote:
>
>> OK, thanks. Are you using the default implementation of the Provenance
>> Repository? I.e., the PersistentProvenanceRepository?
>>
>>
>> On Aug 17, 2018, at 10:10 AM, Daniel Watson <dc...@gmail.com> wrote:
>>
>> Mark,
>>
>> 1.7.0
>> 06/19/2018 21:55:30 EDT
>> Tagged nifi-1.7.0-RC1
>>
>> Thanks
>>
>> On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Hi Daniel,
>>>
>>> What version of NiFi are you running?
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> > On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>
>>> wrote:
>>> >
>>> > Anyone have any issues with the data lineage screen? My NiFi
>>> instance can't compute the data lineage for a specific flow. It worked
>>> originally, then after running some files through, it no longer will. It
>>> gets stuck at 100% and the UI shows nothing.
>>> >
>>> > The flow is...
>>> >
>>> > list files -> update att -> order -> fetch files -> update att -> put
>>> files
>>> >
>>> > I removed the provenance directory and restarted, and it did the same
>>> thing.
>>>
>>>
>>
>
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Mark Payne <ma...@hotmail.com>.
OK, my recommendation would be to change to the WriteAheadProvenanceRepository. You can do this by
changing the value of the "nifi.provenance.repository.implementation" property in nifi.properties to
"org.apache.nifi.provenance.WriteAheadProvenanceRepository"
The WriteAhead implementation was written several releases back, but we left Persistent Provenance Repo
as the default just to ensure that the Write-Ahead implementation was stable enough. At this point, it's pretty
clear that it is not only much faster than the Persistent Prov Repo but also far more stable. There is a JIRA [1]
to make this the new default.
The nice thing is that it is backward compatible, so if you change to the WriteAhead impl and restart, you should
still be able to query all of your existing data without losing anything.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-5482
On Aug 17, 2018, at 10:26 AM, Daniel Watson <dc...@gmail.com>> wrote:
Yes.
On Fri, Aug 17, 2018 at 10:19 AM Mark Payne <ma...@hotmail.com>> wrote:
OK, thanks. Are you using the default implementation of the Provenance Repository? I.e., the PersistentProvenanceRepository?
On Aug 17, 2018, at 10:10 AM, Daniel Watson <dc...@gmail.com>> wrote:
Mark,
1.7.0
06/19/2018 21:55:30 EDT
Tagged nifi-1.7.0-RC1
Thanks
On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com>> wrote:
Hi Daniel,
What version of NiFi are you running?
Thanks
-Mark
> On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>> wrote:
>
> Anyone have any issues with the data lineage screen? My NiFi instance can't compute the data lineage for a specific flow. It worked originally, then after running some files through, it no longer will. It gets stuck at 100% and the UI shows nothing.
>
> The flow is...
>
> list files -> update att -> order -> fetch files -> update att -> put files
>
> I removed the provenance directory and restarted, and it did the same thing.
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Daniel Watson <dc...@gmail.com>.
Yes.
On Fri, Aug 17, 2018 at 10:19 AM Mark Payne <ma...@hotmail.com> wrote:
> OK, thanks. Are you using the default implementation of the Provenance
> Repository? I.e., the PersistentProvenanceRepository?
>
>
> On Aug 17, 2018, at 10:10 AM, Daniel Watson <dc...@gmail.com> wrote:
>
> Mark,
>
> 1.7.0
> 06/19/2018 21:55:30 EDT
> Tagged nifi-1.7.0-RC1
>
> Thanks
>
> On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com> wrote:
>
>> Hi Daniel,
>>
>> What version of NiFi are you running?
>>
>> Thanks
>> -Mark
>>
>>
>> > On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>
>> wrote:
>> >
>> > Anyone have any issues with the data lineage screen? My NiFi instance
>> can't compute the data lineage for a specific flow. It worked originally,
>> then after running some files through, it no longer will. It gets stuck at
>> 100% and the UI shows nothing.
>> >
>> > The flow is...
>> >
>> > list files -> update att -> order -> fetch files -> update att -> put
>> files
>> >
>> > I removed the provenance directory and restarted, and it did the same
>> thing.
>>
>>
>
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Mark Payne <ma...@hotmail.com>.
OK, thanks. Are you using the default implementation of the Provenance Repository? I.e., the PersistentProvenanceRepository?
On Aug 17, 2018, at 10:10 AM, Daniel Watson <dc...@gmail.com>> wrote:
Mark,
1.7.0
06/19/2018 21:55:30 EDT
Tagged nifi-1.7.0-RC1
Thanks
On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com>> wrote:
Hi Daniel,
What version of NiFi are you running?
Thanks
-Mark
> On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>> wrote:
>
> Anyone have any issues with the data lineage screen? My NiFi instance can't compute the data lineage for a specific flow. It worked originally, then after running some files through, it no longer will. It gets stuck at 100% and the UI shows nothing.
>
> The flow is...
>
> list files -> update att -> order -> fetch files -> update att -> put files
>
> I removed the provenance directory and restarted, and it did the same thing.
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Daniel Watson <dc...@gmail.com>.
Mark,
1.7.0
06/19/2018 21:55:30 EDT
Tagged nifi-1.7.0-RC1
Thanks
On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <ma...@hotmail.com> wrote:
> Hi Daniel,
>
> What version of NiFi are you running?
>
> Thanks
> -Mark
>
>
> > On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com>
> wrote:
> >
> > Anyone have any issues with the data lineage screen? My NiFi instance
> can't compute the data lineage for a specific flow. It worked originally,
> then after running some files through, it no longer will. It gets stuck at
> 100% and the UI shows nothing.
> >
> > The flow is...
> >
> > list files -> update att -> order -> fetch files -> update att -> put
> files
> >
> > I removed the provenance directory and restarted, and it did the same
> thing.
>
>
Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Posted by Mark Payne <ma...@hotmail.com>.
Hi Daniel,
What version of NiFi are you running?
Thanks
-Mark
> On Aug 17, 2018, at 10:07 AM, Daniel Watson <dc...@gmail.com> wrote:
>
> Anyone have any issues with the data lineage screen? My NiFi instance can't compute the data lineage for a specific flow. It worked originally, then after running some files through, it no longer will. It gets stuck at 100% and the UI shows nothing.
>
> The flow is...
>
> list files -> update att -> order -> fetch files -> update att -> put files
>
> I removed the provenance directory and restarted, and it did the same thing.