Lineage in custom nodes that aren't directly connected

[gertdetlef <ge...@gmx.de>]

Hi developers,

I have the following setup:
A sender node reads data from a flow file and puts that via messages into a
Kafka topic. It appends the UUID of the flow-file to each message created.
The messages are then processed by someone else and a follow-up message is
created and put back to Kafka (another topic).
A collector node reads data from the response topic in Kafka and creates a
flow file per UUID found in the response. I use
{{session.getProvenanceReporter().fetch(...)}} to report that there is a
newly created flow file. Also, I set the property "parent.uuid" using
{{session.putAttribute( flowFile, "parent.uuid", sourceUuid )}}.

Observations:
Once data is processed and viewing the provenance graph, I observe the
following:
- The sender stops reporting after the message was sent to Kafka
- The collector starts reporting from the Kafka receiving point onwards with
a fetch event
- The parent.uuid information of the collector is correctly displayed within
the collector stream and can be used to manually search within the Nifi UI
for the original flow file. This works.

What I want to achieve is that the flow file sent to Kafka isn't stopping in
the provenance graph -- it shall contain the graph and therewith the created
final flow files of the collector as well. How can I achieve this?

BR

Frank



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Lineage-in-custom-nodes-that-aren-t-directly-connected-tp16739.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.