You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Yi Wang <cs...@gmail.com> on 2021/01/11 14:53:02 UTC
Cross system data lineage
Hi Nifi team and experts,
In such an example scenario:
Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —> Nifi —> s3
How do I fill in the ‘gap’ of data lineage?
Once the data leave Nifi, its provenance life is ended (as far as I know), even when the same data is sent back to Nifi later, Nifi treats them as different data. So how can I handle this in order to get a complete data lineage graph?
Any idea or suggestions? Thanks in advance.
Cheers
Re: Cross system data lineage
Posted by Yi Wang <cs...@gmail.com>.
Hi Chris
For this alternative id solution, when the data is processed by external
system, is there way to generate the provenance event for “processing in
Spark” stage?
Chris Sampson <ch...@naimuri.com>于2021年1月11日 周一下午3:58写道：
> Might be worth taking a look at the "alternate.identifier" (see
> https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#associating-disparate-data
> ).
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
> <https://www.naimuri.com/>
>
>
> On Mon, 11 Jan 2021 at 14:53, Yi Wang <cs...@gmail.com> wrote:
>
>> Hi Nifi team and experts,
>>
>> In such an example scenario:
>>
>> Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —>
>> Nifi —> s3
>>
>> How do I fill in the ‘gap’ of data lineage?
>> Once the data leave Nifi, its provenance life is ended (as far as I
>> know), even when the same data is sent back to Nifi later, Nifi treats them
>> as different data. So how can I handle this in order to get a complete data
>> lineage graph?
>>
>> Any idea or suggestions? Thanks in advance.
>>
>>
>> Cheers
>>
>>
Re: Cross system data lineage
Posted by Yi Wang <cs...@gmail.com>.
Hi Pierre
Atlas indeed comes into my attention in the beginning, but my production
environment does not support hdfs which is the fundamental of Atlas.
The provenance is a out of box feature in NiFi, and we also use NiFi to do
simple transformations, therefore I am searching some provenance solution
that can be leveraged in NiFi, that would be perfect.
Cheers
Pierre Villard <pi...@gmail.com>于2021年1月11日 周一下午6:42写道：
> Hi,
>
> Just want to also mention that the integration that NiFi provides with
> Apache Atlas (through a dedicated reporting task) is supposed to provide
> this kind of end-to-end lineage across multiple components.
>
> Thanks,
> Pierre
>
> Le lun. 11 janv. 2021 à 18:58, Chris Sampson <ch...@naimuri.com>
> a écrit :
>
>> Might be worth taking a look at the "alternate.identifier" (see
>> https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#associating-disparate-data
>> ).
>>
>>
>> ---
>> *Chris Sampson*
>> IT Consultant
>> chris.sampson@naimuri.com
>> <https://www.naimuri.com/>
>>
>>
>> On Mon, 11 Jan 2021 at 14:53, Yi Wang <cs...@gmail.com> wrote:
>>
>>> Hi Nifi team and experts,
>>>
>>> In such an example scenario:
>>>
>>> Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —>
>>> Nifi —> s3
>>>
>>> How do I fill in the ‘gap’ of data lineage?
>>> Once the data leave Nifi, its provenance life is ended (as far as I
>>> know), even when the same data is sent back to Nifi later, Nifi treats them
>>> as different data. So how can I handle this in order to get a complete data
>>> lineage graph?
>>>
>>> Any idea or suggestions? Thanks in advance.
>>>
>>>
>>> Cheers
>>>
>>>
Re: Cross system data lineage
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
Just want to also mention that the integration that NiFi provides with
Apache Atlas (through a dedicated reporting task) is supposed to provide
this kind of end-to-end lineage across multiple components.
Thanks,
Pierre
Le lun. 11 janv. 2021 à 18:58, Chris Sampson <ch...@naimuri.com> a
écrit :
> Might be worth taking a look at the "alternate.identifier" (see
> https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#associating-disparate-data
> ).
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
> <https://www.naimuri.com/>
>
>
> On Mon, 11 Jan 2021 at 14:53, Yi Wang <cs...@gmail.com> wrote:
>
>> Hi Nifi team and experts,
>>
>> In such an example scenario:
>>
>> Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —>
>> Nifi —> s3
>>
>> How do I fill in the ‘gap’ of data lineage?
>> Once the data leave Nifi, its provenance life is ended (as far as I
>> know), even when the same data is sent back to Nifi later, Nifi treats them
>> as different data. So how can I handle this in order to get a complete data
>> lineage graph?
>>
>> Any idea or suggestions? Thanks in advance.
>>
>>
>> Cheers
>>
>>
Re: Cross system data lineage
Posted by Chris Sampson <ch...@naimuri.com>.
Might be worth taking a look at the "alternate.identifier" (see
https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#associating-disparate-data
).
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
<https://www.naimuri.com/>
On Mon, 11 Jan 2021 at 14:53, Yi Wang <cs...@gmail.com> wrote:
> Hi Nifi team and experts,
>
> In such an example scenario:
>
> Data —> Nifi —> Kafka —> external system (spark, flink etc) —> Kafka —>
> Nifi —> s3
>
> How do I fill in the ‘gap’ of data lineage?
> Once the data leave Nifi, its provenance life is ended (as far as I know),
> even when the same data is sent back to Nifi later, Nifi treats them as
> different data. So how can I handle this in order to get a complete data
> lineage graph?
>
> Any idea or suggestions? Thanks in advance.
>
>
> Cheers
>
>