You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Herman Yu <he...@teeupdata.com> on 2016/01/15 16:24:37 UTC

Hive column level lineage

Hi Everyone,

I think the current lineage supporting with Hive is at Hive table level, is there a plan to enhance the hive bridge and extend the lineage support to Hive column level?

Hive column level lineage can be built with customized logic through API calls and with “Process”. However, the built-in hive_column entity is NOT inherited from Dataset, is there a specific reason? 

Thanks
Herman.

Re: Hive column level lineage

Posted by Shwetha Shivalingamurthy <ss...@hortonworks.com>.

https://issues.apache.org/jira/browse/ATLAS-247 - the one that you had
created


Regards,
Shwetha






On 18/01/16 6:57 pm, "Herman Yu" <he...@teeupdata.com> wrote:

>Thanks Shwetha.  Is there a Jira item tracking this feature?  Herman.
>
>> On Jan 18, 2016, at 1:13 AM, Shwetha Shivalingamurthy
>><ss...@hortonworks.com> wrote:
>> 
>> Hi Herman,
>> 
>> We will need the query plan from hive to be able to implement hive
>>column
>> lineage. Once we have the query plan, we have to modify the hive model
>>to
>> implement column level lineage in Atlas. We are targetting this for the
>> next Atlas release.
>> 
>> Regards,
>> Shwetha
>> 
>> 
>> 
>> 
>> 
>> 
>> On 15/01/16 8:54 pm, "Herman Yu" <he...@teeupdata.com> wrote:
>> 
>>> Hi Everyone,
>>> 
>>> I think the current lineage supporting with Hive is at Hive table
>>>level,
>>> is there a plan to enhance the hive bridge and extend the lineage
>>>support
>>> to Hive column level?
>>> 
>>> Hive column level lineage can be built with customized logic through
>>>API
>>> calls and with ³Process². However, the built-in hive_column entity is
>>>NOT
>>> inherited from Dataset, is there a specific reason?
>>> 
>>> Thanks
>>> Herman.
>>> 
>>> 
>>> 
>> 
>
>

Re: Hive column level lineage

Posted by Herman Yu <he...@teeupdata.com>.

Thanks Shwetha.  Is there a Jira item tracking this feature?  Herman.

> On Jan 18, 2016, at 1:13 AM, Shwetha Shivalingamurthy <ss...@hortonworks.com> wrote:
> 
> Hi Herman,
> 
> We will need the query plan from hive to be able to implement hive column
> lineage. Once we have the query plan, we have to modify the hive model to
> implement column level lineage in Atlas. We are targetting this for the
> next Atlas release.
> 
> Regards,
> Shwetha
> 
> 
> 
> 
> 
> 
> On 15/01/16 8:54 pm, "Herman Yu" <he...@teeupdata.com> wrote:
> 
>> Hi Everyone,
>> 
>> I think the current lineage supporting with Hive is at Hive table level,
>> is there a plan to enhance the hive bridge and extend the lineage support
>> to Hive column level?
>> 
>> Hive column level lineage can be built with customized logic through API
>> calls and with ³Process². However, the built-in hive_column entity is NOT
>> inherited from Dataset, is there a specific reason?
>> 
>> Thanks
>> Herman.
>> 
>> 
>> 
>

Re: Hive column level lineage

Posted by Shwetha Shivalingamurthy <ss...@hortonworks.com>.

Hi Herman,

We will need the query plan from hive to be able to implement hive column
lineage. Once we have the query plan, we have to modify the hive model to
implement column level lineage in Atlas. We are targetting this for the
next Atlas release.

Regards,
Shwetha

On 15/01/16 8:54 pm, "Herman Yu" <he...@teeupdata.com> wrote:

>Hi Everyone,
>
>I think the current lineage supporting with Hive is at Hive table level,
>is there a plan to enhance the hive bridge and extend the lineage support
>to Hive column level?
>
>Hive column level lineage can be built with customized logic through API
>calls and with ³Process². However, the built-in hive_column entity is NOT
>inherited from Dataset, is there a specific reason?
>
>Thanks
>Herman.
>
>
>