You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org> on 2026/02/20 23:46:03 UTC

[Impala-ASF-CR] IMPALA-14768: [WIP] Add operation type to the lineage graph

Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24018


Change subject: IMPALA-14768: [WIP] Add operation type to the lineage graph
......................................................................

IMPALA-14768: [WIP] Add operation type to the lineage graph

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
3 files changed, 8 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Peter Rozsa, Daniel Vanko, Steve Carlin, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#5).

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

Testing:
 - Added a new test case to lineage.test to show that Impala could
   produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test to make sure each lineage event comes with its
   respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatement.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatementImpl.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
12 files changed, 393 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 13:03:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 5:

(1 comment)

I have addressed Zoltan's comment on patch set 4. Let me know if there are additional suggestions. Thanks!

http://gerrit.cloudera.org:8080/#/c/24018/4/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/4/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@982
PS4, Line 982: turn OperationType.CREATEVIEW;
> instanceof could be better here as it would handle subclasses as well if th
Thanks for pointing this out Zoltan!

Indeed, for queries like the following, operation type would be UNKNOWN if we do not use instanceof.

 create table lineage_test_db.like_alltypestiny_file like parquet 
 'hdfs:///test-warehouse/schemas/alltypestiny.parquet';

I will make the corresponding change in the next patch and will add a new test case to lineage.test too.



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 19:53:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: [WIP] Add operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: [WIP] Add operation type to the lineage graph
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21738/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 21 Feb 2026 00:23:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13157/


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 03:09:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21812/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 23:15:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4: Code-Review+1

(1 comment)

LGTM!

http://gerrit.cloudera.org:8080/#/c/24018/4/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/4/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@982
PS4, Line 982: stmtClass.equals(CreateViewStmt.class)
instanceof could be better here as it would handle subclasses as well if there's any.



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 16:07:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13196/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 13:03:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21833/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 20:19:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Peter Rozsa, Daniel Vanko, Steve Carlin, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#7).

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

On the other hand, apart from the field of 'operationType_', this
patch also makes PlannerTest#testLineage() check the field of
'queryStr_' of ColumnLineageGraph when testLineage() compares the
actual lineage graph with the expected one in lineage.test run in the
frontend test.

Testing:
 - Added a new test case to lineage.test run in end-to-end test to show
   that Impala could produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test run in end-to-end and frontend tests to make
   sure each lineage event comes with its respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatement.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatementImpl.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
14 files changed, 458 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 7
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: [WIP] Add operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: [WIP] Add operation type to the lineage graph
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21794/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 00:49:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

On the other hand, apart from the field of 'operationType_', this
patch also makes PlannerTest#testLineage() check the field of
'queryStr_' of ColumnLineageGraph when testLineage() compares the
actual lineage graph with the expected one in lineage.test run in the
frontend test.

Testing:
 - Added a new test case to lineage.test run in end-to-end test to show
   that Impala could produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test run in end-to-end and frontend tests to make
   sure each lineage event comes with its respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Reviewed-on: http://gerrit.cloudera.org:8080/24018
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatement.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatementImpl.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
14 files changed, 458 insertions(+), 14 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 9
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#4).

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

Testing:
 - Added a new test case to lineage.test to show that Impala could
   produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test to make sure each lineage event comes with its
   respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatement.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatementImpl.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
12 files changed, 378 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21821/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 01:08:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 04:54:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 7:

(1 comment)

I have addressed Daniel's comment on patch set 6. Let me know if there are additional suggestions. Thanks!

http://gerrit.cloudera.org:8080/#/c/24018/6/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/6/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@910
PS6, Line 910: .queryStr_)) return false
> Use g instead from previous line.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 7
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 18:16:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 7
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 18:38:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 7
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 13:02:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Daniel Vanko has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 6:

(1 comment)

Apart from this one comment, LGTM!

http://gerrit.cloudera.org:8080/#/c/24018/6/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/6/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@910
PS6, Line 910: (ColumnLineageGraph) obj)
Use g instead from previous line.



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 09:24:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Daniel Vanko has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 5: Code-Review-1

(4 comments)

Thanks for working on this, great addition to the lineage output!

However, there's also another lineage.test file, used by PlannerTest. I think that should also include operationType and needs some ground-working as well.

http://gerrit.cloudera.org:8080/#/c/24018/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24018/5//COMMIT_MSG@42
PS5, Line 42:  - Updated lineage.test to make sure each lineage event comes with its
            :    respective operation type.
            : 
We have another lineage.test file at testdata/workloads/functional-planner/queries/PlannerTest/lineage.test, I think it also need to be updated.
Moreover, PlannerTestBase's checkColumnLineage https://github.com/apache/impala/blob/79e2ab99/fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java#L775 uses ColumnLineageGraph.createFromJSON and ColumnLineageGraph.fromThrift, so those also should be extended to handle operation type.


http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@814
PS5, Line 814:   public static ColumnLineageGraph fromThrift(TLineageGraph obj) {
Add operationType handling.


http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@841
PS5, Line 841:   public static ColumnLineageGraph createFromJSON(String json) {
Add operationType handling.


http://gerrit.cloudera.org:8080/#/c/24018/5/testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
File testdata/workloads/functional-planner/queries/PlannerTest/lineage.test:

PS5: 
Here is the other lineage.test, which should also include operationType.



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 14:18:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: [WIP] Add operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#2).

Change subject: IMPALA-14768: [WIP] Add operation type to the lineage graph
......................................................................

IMPALA-14768: [WIP] Add operation type to the lineage graph

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
8 files changed, 21 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13166/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 00:40:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4:

In patch set 4, I fixed a bug due to which we could pass a null value (ctx_.getAnalysisResult().getStmt()) to the last argument of computeLineageGraph() (https://gerrit.cloudera.org/c/24018/3..4/fe/src/main/java/org/apache/impala/planner/Planner.java#b283). This is the case when Calcite is the planner.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 00:44:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 4:

Hi all, please let me know if there is any comment on the patch. Thanks very much for the help!


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 05:49:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Peter Rozsa, Daniel Vanko, Steve Carlin, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#6).

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

On the other hand, apart from the field of 'operationType_', this
patch also makes PlannerTest#testLineage() check the field of
'queryStr_' of ColumnLineageGraph when testLineage() compares the
actual lineage graph with the expected one in lineage.test run in the
frontend test.

Testing:
 - Added a new test case to lineage.test run in end-to-end test to show
   that Impala could produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test run in end-to-end and frontend tests to make
   sure each lineage event comes with its respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatement.java
M fe/src/main/java/org/apache/impala/analysis/ParsedStatementImpl.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
14 files changed, 458 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 17:18:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24018

to look at the new patch set (#3).

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................

IMPALA-14768: Add the operation type to the lineage graph

This patch makes Impala produce the operation type of the completed
query in the corresponding lineage event so that it would be easier for
data lineage tools like Apache Atlas to derive the operation type of a
given query. Note that currently Apache Atlas determines the operation
type of a given Impala query by matching the field of 'queryText' in the
lineage event against predefined regular expressions. Refer to
https://github.com/apache/atlas/blob/2957ff2/addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java#L49-L77
for more details.

However, such an approach is not robust. Recall that the string in
'queryText' is produced by Impala server replacing each newline in the
original query string with a space, which is followed by redaction.
Thus, 'queryText' may not be a valid SQL statement afterward.

  string stmt =
    replace_all_copy(query_ctx->client_request.stmt, "\n", " ");
  Redact(&stmt);
  // 'redacted_stmt' will be the string Impala uses to populate
  // 'queryText' of the lineage event.
  query_ctx->client_request.__set_redacted_stmt((const string) stmt);

For instance, when the original query
string contains a one-line SQL comment, it could be difficult for one to
decide where that one-line SQL comment ends if every newline in the
original query string is already replaced with a space.

Therefore, after this patch, it would be much easier for data lineage
tools to determine the operation type since it will be directly provided
in the lineage log.

Testing:
 - Added a new test case to lineage.test to show that Impala could
   produce a lineage event for INSERT OVERWRITE.
 - Updated lineage.test to make sure each lineage event comes with its
   respective operation type.

Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
---
M be/src/util/lineage-util.h
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M testdata/workloads/functional-query/queries/QueryTest/lineage.test
9 files changed, 333 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24018/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13157/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 22:49:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 6:

(4 comments)

In patch set 6, I have addressed Daniel's comments on patch set 5 by updating the golden file of the front-end lineage test.

Let me know if there are additional suggestions. Thanks!

http://gerrit.cloudera.org:8080/#/c/24018/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24018/5//COMMIT_MSG@42
PS5, Line 42: actual lineage graph with the expected one in lineage.test run in the
            : frontend test.
            : 
> We have another lineage.test file at testdata/workloads/functional-planner/
Thanks Daniel for pointing this out!

You are correct. I will revise patch set 5 accordingly.


http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
File fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java:

http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@814
PS5, Line 814: 
> Add operationType handling.
Done


http://gerrit.cloudera.org:8080/#/c/24018/5/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java@841
PS5, Line 841: 
> Add operationType handling.
Done


http://gerrit.cloudera.org:8080/#/c/24018/5/testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
File testdata/workloads/functional-planner/queries/PlannerTest/lineage.test:

PS5: 
> Here is the other lineage.test, which should also include operationType.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 00:12:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14768: Add the operation type to the lineage graph

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24018 )

Change subject: IMPALA-14768: Add the operation type to the lineage graph
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21853/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24018
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb94120a9bb1b994d4e681ea98521035bcc6510e
Gerrit-Change-Number: 24018
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 00:31:58 +0000
Gerrit-HasComments: No