[jira] [Commented] (HIVE-11139) Emit more lineage information

["Mark Grover (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745937#comment-14745937 ] 

Mark Grover commented on HIVE-11139:
------------------------------------

I have a dynamic partitioning query but at the end of the query it shows me an error message like:
{quote}
ERROR : Result schema has 2 fields, but we don't get as many dependencies
{quote}

Going through the source code, led me to this commit. Was this tested to make sure it works fine with dynamic partitioning. Here's my query btw?
{code}
SET hive.exec.dynamic.partition.mode=nonstrict;
DROP TABLE IF EXISTS default.src_mark;
CREATE TABLE default.src_mark (first string, word string)
PARTITIONED BY (length int)
STORED AS PARQUET;
INSERT INTO TABLE default.src_mark PARTITION(length) SELECT first, word, length FROM spark_hive.src_flat;
{code}

And, I verified that all the values in src_flat conform to the schema. Also, at the very least it would be helpful to know what the number of dependencies and what their names were in the error message:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java#L251

Your thoughts would be much appreciated!

> Emit more lineage information
> -----------------------------
>
>                 Key: HIVE-11139
>                 URL: https://issues.apache.org/jira/browse/HIVE-11139
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Jimmy Xiang
>            Assignee: Jimmy Xiang
>             Fix For: 1.3.0, 2.0.0
>
>         Attachments: HIVE-11139.1.patch, HIVE-11139.2.patch, HIVE-11139.3.patch
>
>
> HIVE-1131 emits some column lineage info. But it doesn't support INSERT statements, or CTAS statements. It doesn't emit the predicate information either.
> We can enhance and use the dependency information created in HIVE-1131, generate more complete lineage info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)