You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by "r7raul1984@163.com" <r7...@163.com> on 2015/06/12 02:23:42 UTC
some advice to tez ui
I think tez ui can add some feature like data governance capabilities such as - which source table generate the result table,provide lineage search for dag.
r7raul1984@163.com
Re: Re: some advice to tez ui
Posted by "r7raul1984@163.com" <r7...@163.com>.
And what do you mean lineage search for dag ? I mean user can find ins and outs of the sql by table name.
For example:
INSERT overwrite TABLE TESTTMP.cust_sku_vis_detl1 SELECT
......
FROM
DW.fct_ordr_detl detl
JOIN DW.fct_ordr ordr ON detl.ordr_id = ordr.ordr_id
AND detl.ds = ordr.ds
JOIN DW.dim_prod prod ON detl.prod_id = prod.prod_id
AND prod.cur_flag = 1
JOIN DW.hier_categ categ ON prod.categ_lvl_id = categ.categ_lvl_id
AND categ.cur_flag = 1
WHERE
detl.ds = '2015-06-01'
AND ordr.ds = '2015-06-01'
GROUP BY
ordr.ds,
ordr.end_user_id,
categ.categ_lvl1_id,
categ.categ_lvl1_name
Tez UI can pick out some sql (dag graph) when user search for TESTTMP.cust_sku_vis_detl1.
r7raul1984@163.com
From: Jeff Zhang
Date: 2015-06-12 08:38
To: user
Subject: Re: some advice to tez ui
This might need hive add some app specific info on the dag/vertex. Actually Tez don't know the concept of table, tez use InputDescriptor/OutputDescriptor to represent the Input/Output which is not restricted to hive table, it could be any data source.
And what do you mean lineage search for dag ?
On Fri, Jun 12, 2015 at 8:23 AM, r7raul1984@163.com <r7...@163.com> wrote:
I think tez ui can add some feature like data governance capabilities such as - which source table generate the result table,provide lineage search for dag.
r7raul1984@163.com
--
Best Regards
Jeff Zhang
Re: some advice to tez ui
Posted by Jeff Zhang <zj...@gmail.com>.
This might need hive add some app specific info on the dag/vertex. Actually
Tez don't know the concept of table, tez use
InputDescriptor/OutputDescriptor to represent the Input/Output which is not
restricted to hive table, it could be any data source.
And what do you mean lineage search for dag ?
On Fri, Jun 12, 2015 at 8:23 AM, r7raul1984@163.com <r7...@163.com>
wrote:
>
> I think tez ui can add some feature like data governance capabilities such
> as - which source table generate the result table,provide lineage search
> for dag.
> ------------------------------
> r7raul1984@163.com
>
--
Best Regards
Jeff Zhang
Re: some advice to tez ui
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> I think tez ui can add some feature like data governance capabilities
>such as - which source table generate the result table,provide lineage
>search for dag.
Tez is an execution engine and the wrong layer to collect this sort of
information.
If those are your actual requirements (for hive), checkout Apache Atlas -
http://wiki.apache.org/incubator/AtlasProposal
Cheers,
Gopal