You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Frank Maritato <fr...@lootcrate.com> on 2017/03/07 19:14:46 UTC
Data Provenance Question
Hi All,
I'm using Nifi-1.1.1 and I'm seeing issues with having up to date data
provenance events in the UI. If I look in the provenance_repository
directory, the data is definitely there. I'm able to see it with gunzip -c
but it either takes a really long time to show up in the ui (sometimes 5-10
minutes) or doesn't show up at all.
Is this a known issue?
Thanks,
--
Frank Maritato
Data Architect
Re: Data Provenance Question
Posted by Mark Payne <ma...@hotmail.com>.
Hey Frank,
So the way that the provenance repository works, it writes out the data "inline" as the FlowFiles
traverse the system. It then periodically (by default it's every 30 seconds or after writing 100 MB of Provenance data)
"rolls over." When it rolls over, it begins writing data to a new file and starts to index the events
for the old file in Lucene. The events are not available in the UI until they have been indexed into
Lucene. So this could perhaps be a few minutes, depending on your system and how many Provenance
Events are getting generated. If interested, there's a Wiki page that I wrote [1] that goes into much
greater detail.
That being said, the existing implementation is a lot slower than it could be. So in the next release
of NiFi, there is an alternate implementation that can be used. The new implementation is much faster
and will also index the Provenance Events inline so that they are immediately available. [2]
However, you did say "takes a really long time to show up or doesn't show up at all." If they events
do not show up at all, then that certainly sounds like a bug that needs to be looked into more. Can
you find a way to replicate such an issue, where the events don't show up at all?
Thanks
-Mark
[1] https://cwiki.apache.org/confluence/display/NIFI/Persistent+Provenance+Repository+Design
[2] https://issues.apache.org/jira/browse/NIFI-3356
On Mar 7, 2017, at 3:54 PM, Frank Maritato <fr...@lootcrate.com>> wrote:
Hmm I'm not sure if this is the case for me or not. I do copy and paste processors, not process groups. Let me see if I can reproduce this doing that.
On Tue, Mar 7, 2017 at 11:33 AM, Carlos Manuel Fernandes (DSI) <ca...@cgd.pt>> wrote:
Hello Frank,
I had this issue with Nifi-1.1.1 when copied a processor group from one location to another. When I create a new processor group from scratch and copy just the processors inside the problem disappear. The problem, for me, just emerged in copied processor groups. Its your problem too ?
Carlos Fernandes
From: Frank Maritato [mailto:frank.maritato@lootcrate.com<ma...@lootcrate.com>]
Sent: terça-feira, 7 de Março de 2017 19:15
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Data Provenance Question
Hi All,
I'm using Nifi-1.1.1 and I'm seeing issues with having up to date data provenance events in the UI. If I look in the provenance_repository directory, the data is definitely there. I'm able to see it with gunzip -c but it either takes a really long time to show up in the ui (sometimes 5-10 minutes) or doesn't show up at all.
Is this a known issue?
Thanks,
--
Frank Maritato
Data Architect
--
Frank Maritato
Data Architect
Re: Data Provenance Question
Posted by Frank Maritato <fr...@lootcrate.com>.
Hmm I'm not sure if this is the case for me or not. I do copy and paste
processors, not process groups. Let me see if I can reproduce this doing
that.
On Tue, Mar 7, 2017 at 11:33 AM, Carlos Manuel Fernandes (DSI) <
carlos.antonio.fernandes@cgd.pt> wrote:
> Hello Frank,
>
>
>
> I had this issue with Nifi-1.1.1 when copied a processor group from one
> location to another. When I create a new processor group from scratch and
> copy just the processors inside the problem disappear. The problem, for
> me, just emerged in copied processor groups. Its your problem too ?
>
>
>
> Carlos Fernandes
>
>
>
>
>
> *From:* Frank Maritato [mailto:frank.maritato@lootcrate.com]
> *Sent:* terça-feira, 7 de Março de 2017 19:15
> *To:* users@nifi.apache.org
> *Subject:* Data Provenance Question
>
>
>
> Hi All,
>
>
>
> I'm using Nifi-1.1.1 and I'm seeing issues with having up to date data
> provenance events in the UI. If I look in the provenance_repository
> directory, the data is definitely there. I'm able to see it with gunzip -c
> but it either takes a really long time to show up in the ui (sometimes 5-10
> minutes) or doesn't show up at all.
>
>
>
> Is this a known issue?
>
>
>
> Thanks,
>
> --
>
> Frank Maritato
>
> Data Architect
>
--
Frank Maritato
Data Architect
RE: Data Provenance Question
Posted by "Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt>.
Hello Frank,
I had this issue with Nifi-1.1.1 when copied a processor group from one location to another. When I create a new processor group from scratch and copy just the processors inside the problem disappear. The problem, for me, just emerged in copied processor groups. Its your problem too ?
Carlos Fernandes
From: Frank Maritato [mailto:frank.maritato@lootcrate.com]
Sent: terça-feira, 7 de Março de 2017 19:15
To: users@nifi.apache.org
Subject: Data Provenance Question
Hi All,
I'm using Nifi-1.1.1 and I'm seeing issues with having up to date data provenance events in the UI. If I look in the provenance_repository directory, the data is definitely there. I'm able to see it with gunzip -c but it either takes a really long time to show up in the ui (sometimes 5-10 minutes) or doesn't show up at all.
Is this a known issue?
Thanks,
--
Frank Maritato
Data Architect