Lost Trace of FowFiles only with attributes

[Paresh Shah <Pa...@lifelock.com>]

In the cluster mode we have seeing the following two issues.

  1.  In one scenario we see that the flowFile( only containing attributes ) is sent over to the remote cluster and after the first processor following the InputPort in the pipeline the data provnance shows no trace of that flow file. Where can we locate this flow file in the repository?

2. In another scenario we see that the data provenance on the receiving cluster is not showing any updates since 4/5 hours even though the sending cluster indicates  that it has successfully sent data to it. So we are not sure where the data is dropped.

We have checked that the clocks on all the cluster nodes are in sync.

Paresh


________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________

Re: Lost Trace of FowFiles only with attributes

[Matthew Clarke <ma...@gmail.com>]

Paresh,

Question 1:
        Not sure I am following exactly what you are saying.  Not every
processor in NiFi will produce a provenance event. Provenance events are
created when an actionable event has occurred against a  FlowFile (receive,
route, modify, drop, etc...).  For example the logAttribute processor would
not normally result in a Provenance event with the exception of when the
success relationship is auto-terminated (drop event).
        Just to make sure I understand.  How are you tracing the file
through Provenance?  On the sending/source NiFi Provenance will be able to
trace the file all the way to delivery to target system.  That transfer
will be the last event recorded which will show the flowfile being sent to
a particular input port on s specific target NiFi instance.  To continue
tracing the lineage of this Flowfile you will need to another provenance
search on the receiving/target NiFi.

Question 2:
        Are you seeing an error messages in the app.log of the receiving
NiFi?  Anything related to permission issues preventing the writing of
events to the Provenance repo?  Are Provenance events created if you
generate FlowFiles locally on that receiving system?  A simple flow of
GenerateFlowFile ---> updateAttribute (Auto-terminate success) should
create provenance events (CREATE, ATTRIBUTES_MODIFIED, and DROP).  If
events are still not created in that system provenance, the issue is not
related to S2S but rather something local to that system/cluster.
        Does the sending NiFi Provenance show the successful delivery of
Data to sending NiFi?  Can you share the details of one of those provenance
SEND events with us?

Thanks,
Matt

On Thu, Feb 4, 2016 at 12:32 PM, Paresh Shah <Pa...@lifelock.com>
wrote:

> In the cluster mode we have seeing the following two issues.
>
>   1.  In one scenario we see that the flowFile( only containing attributes
> ) is sent over to the remote cluster and after the first processor
> following the InputPort in the pipeline the data provnance shows no trace
> of that flow file. Where can we locate this flow file in the repository?
>
> 2. In another scenario we see that the data provenance on the receiving
> cluster is not showing any updates since 4/5 hours even though the sending
> cluster indicates  that it has successfully sent data to it. So we are not
> sure where the data is dropped.
>
> We have checked that the clocks on all the cluster nodes are in sync.
>
> Paresh
>
>
> ________________________________
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ________________________________
>