You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by wenhemin <wh...@163.com> on 2020/04/15 06:57:46 UTC
[DISCUSS] Data lineage (Table Level)
hello, friends!
I plan to implement data lineage function (Table level). Solicit your opinions.
The sql node and etl node automatically parse the dependency table and target table.
The frontend controls whether to enable dependency detection through switch.
The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
Rely on the node to set the default number of retries.
Open the node that dependent detection function, no longer need to manually connect.
Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>
Re:[DISCUSS] Data lineage (Table Level)
Posted by "xingchun.chen" <xi...@qq.com>.
good job,I look forward to your contribution
------------------ Original ------------------
From: "wenhemin"<whm_777@163.com>;
Date: Wed, Apr 15, 2020 02:57 PM
To: "dev"<dev@dolphinscheduler.apache.org>;
Subject: [DISCUSS] Data lineage (Table Level)
hello, friends!
I plan to implement data lineage function (Table level). Solicit your opinions.
The sql node and etl node automatically parse the dependency table and target table.
The frontend controls whether to enable dependency detection through switch.
The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
Rely on the node to set the default number of retries.
Open the node that dependent detection function, no longer need to manually connect.
Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>
Re: [DISCUSS] Data lineage (Table Level)
Posted by lidong dai <da...@gmail.com>.
good job
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
wenhemin <wh...@163.com> 于2020年4月15日周三 下午2:58写道：
> hello, friends!
>
> I plan to implement data lineage function (Table level). Solicit your
> opinions.
>
> The sql node and etl node automatically parse the dependency table and
> target table.
> The frontend controls whether to enable dependency detection through
> switch.
> The master server automatically injects dependent nodes, create dependent
> nodes based on dependencies.
> Rely on the node to set the default number of retries.
> Open the node that dependent detection function, no longer need to
> manually connect.
>
> Issue link :
> https://github.com/apache/incubator-dolphinscheduler/issues/249 <
> https://github.com/apache/incubator-dolphinscheduler/issues/249>
Re: [DISCUSS] Data lineage (Table Level)
Posted by wenhemin <wh...@163.com>.
Yes, A new page. A new button is added to the operation column of The workflow definition list page.
Convenient to view the relationship between nodes in real time.
> 在 2020年4月16日，16:01，lgcareer2019@outlook.com 写道：
>
> Thanks!
> I see the third image is the tree-view page. Your answer is add a dependency graph page,Is this page a new page? or it means the tree-view page.
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： wenhemin
> 发送时间： 2020-04-16 15:15
> 收件人： dev
> 主题： Re: [DISCUSS] Data lineage (Table Level)
> The front-end workflow definition list will add a dependency graph page. You can see the third picture in the issue.
> The virtual depend task does not persist in db.
>
>> 在 2020年4月16日，14:31，lgcareer2019@outlook.com 写道：
>>
>> Sounds great!I It's a good job.
>> I read the issue and have a question about [1].
>>
>> Do the virtual depend nodes can't display in frontend?
>> The visurula depend tasks will be generated when the master process the sql task or etl task,Do they will persisit in db?
>>
>>
>> 1.The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
>> Rely on the node to set the default number of retries.
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> From: wenhemin
>> Date: 2020-04-15 14:57
>> To: dev
>> Subject: [DISCUSS] Data lineage (Table Level)
>> hello, friends!
>>
>> I plan to implement data lineage function (Table level). Solicit your opinions.
>>
>> The sql node and etl node automatically parse the dependency table and target table.
>> The frontend controls whether to enable dependency detection through switch.
>> The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
>> Rely on the node to set the default number of retries.
>> Open the node that dependent detection function, no longer need to manually connect.
>>
>> Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>
>
>
回复: Re: [DISCUSS] Data lineage (Table Level)
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
Thanks!
I see the third image is the tree-view page. Your answer is add a dependency graph page,Is this page a new page? or it means the tree-view page.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： wenhemin
发送时间： 2020-04-16 15:15
收件人： dev
主题： Re: [DISCUSS] Data lineage (Table Level)
The front-end workflow definition list will add a dependency graph page. You can see the third picture in the issue.
The virtual depend task does not persist in db.
> 在 2020年4月16日，14:31，lgcareer2019@outlook.com 写道：
>
> Sounds great!I It's a good job.
> I read the issue and have a question about [1].
>
> Do the virtual depend nodes can't display in frontend?
> The visurula depend tasks will be generated when the master process the sql task or etl task,Do they will persisit in db?
>
>
> 1.The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
> Rely on the node to set the default number of retries.
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: wenhemin
> Date: 2020-04-15 14:57
> To: dev
> Subject: [DISCUSS] Data lineage (Table Level)
> hello, friends!
>
> I plan to implement data lineage function (Table level). Solicit your opinions.
>
> The sql node and etl node automatically parse the dependency table and target table.
> The frontend controls whether to enable dependency detection through switch.
> The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
> Rely on the node to set the default number of retries.
> Open the node that dependent detection function, no longer need to manually connect.
>
> Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>
Re: [DISCUSS] Data lineage (Table Level)
Posted by wenhemin <wh...@163.com>.
The front-end workflow definition list will add a dependency graph page. You can see the third picture in the issue.
The virtual depend task does not persist in db.
> 在 2020年4月16日，14:31，lgcareer2019@outlook.com 写道：
>
> Sounds great!I It's a good job.
> I read the issue and have a question about [1].
>
> Do the virtual depend nodes can't display in frontend?
> The visurula depend tasks will be generated when the master process the sql task or etl task,Do they will persisit in db?
>
>
> 1.The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
> Rely on the node to set the default number of retries.
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: wenhemin
> Date: 2020-04-15 14:57
> To: dev
> Subject: [DISCUSS] Data lineage (Table Level)
> hello, friends!
>
> I plan to implement data lineage function (Table level). Solicit your opinions.
>
> The sql node and etl node automatically parse the dependency table and target table.
> The frontend controls whether to enable dependency detection through switch.
> The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
> Rely on the node to set the default number of retries.
> Open the node that dependent detection function, no longer need to manually connect.
>
> Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>
Re: [DISCUSS] Data lineage (Table Level)
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
Sounds great!I It's a good job.
I read the issue and have a question about [1].
Do the virtual depend nodes can't display in frontend?
The visurula depend tasks will be generated when the master process the sql task or etl task,Do they will persisit in db?
1.The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
Rely on the node to set the default number of retries.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: wenhemin
Date: 2020-04-15 14:57
To: dev
Subject: [DISCUSS] Data lineage (Table Level)
hello, friends!
I plan to implement data lineage function (Table level). Solicit your opinions.
The sql node and etl node automatically parse the dependency table and target table.
The frontend controls whether to enable dependency detection through switch.
The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
Rely on the node to set the default number of retries.
Open the node that dependent detection function, no longer need to manually connect.
Issue link : https://github.com/apache/incubator-dolphinscheduler/issues/249 <https://github.com/apache/incubator-dolphinscheduler/issues/249>