Extract column and table lineage from flink sql

[dawangli <da...@163.com>]

I want to build a lineage system for a real-time data warehouse，how can I extract table and column lineage from flink sql?





 

Re: Extract column and table lineage from flink sql

[Danny Chan <da...@apache.org>]

Hi, dawangli ~

Usually people build the lineage of tables through a self-built platform,
there was a DB to persist the relationship between the tables, for each
job, you may need to analyze each SQL which are source tables and which are
sink.

E.G. The INSERT target table is a sink and table after the scan or join is
a source.

If you got the rel tree, you can get the info by a shuttle, if an AST
instead, you can have a SqlVisitor.

Danny Chan <da...@apache.org> 于2020年10月22日周四 下午5:24写道：

> Hi, dawangli ~
>
> Usually people build the lineage of tables through a self-built platform,
> there was a DB to persist the relationship between the tables, for each
> job, you may need to analyze each SQL which are source tables and which are
> sink.
>
> E.G. The INSERT target table is a sink and table after the scan or join is
> a source.
>
> If you got the rel tree, you can get the info by a shuttle, if an AST
> instead, you can have a SqlVisitor.
>
> dawangli <da...@163.com> 于2020年10月20日周二 下午9:46写道：
>
>> I want to build a lineage system for a real-time data warehouse，how can I
>> extract table and column lineage from flink sql?
>>
>>
>>
>>
>>
>>
>>
>