You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2025/05/08 21:18:47 UTC

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22869


Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................

IMPALA-14051: Use batch insert HMS API to fire insert events

Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch(). Right now,
Impala is using a single call for each partition, so firing insert
statements on a huge partitioned table can become a performance
bottleneck.

This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.

Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 26 insertions(+), 18 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468:       hiveClient.addWriteNotificationLogInBatch(rqst);
Was this a client-side only addition? Or does it change the required HMS server version Impala works with?



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:27:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 03:59:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:44:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG@10
PS2, Line 10: HMS API addWriteNotificationLogInBatch().
"HMS API addWriteNotificationLogInBatch() available starting from Hive 4."

If you can also mention the Hive JIRA number, that will be great.


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@440
PS2, Line 440: Fires a listener event of the type ACID_WRITE on a transactional table in metastore.
Is this improvement limited to Hive ACID table? If yes, please mention it in commit message.


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@457
PS2, Line 457: rqst
nit: "req" is better.


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@466
PS2, Line 466: rqst
nit: "batchedReq" is better.



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Jun 2025 22:44:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3: Code-Review+2

(1 comment)

Carry Michael's +1.
Please monitor downstream exhaustive tests for unforeseen cases.

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> Tests ran green. Are we good to merge this?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:08:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

(1 comment)

@Riza

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> test_events_custom_configs.py#test_event_batching is the right test. But le
Tests ran green. Are we good to merge this?



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:04:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
The partition value is added without any proper delimiter, curious what it would look like in the later debug logging



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 02 Jun 2025 23:03:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19370/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 05:35:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468:       hiveClient.addWriteNotificationLogInBatch(rqst);
> We also build Impala with Hive 3.1.3. I guess if that compiles we're fine, 
This MetastoreShim is used by hive 4. MetastoreShim used by hive-3 doesn't support this method: https://github.com/apache/impala/blob/master/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L385-L386



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 May 2025 23:57:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468:       hiveClient.addWriteNotificationLogInBatch(rqst);
> This is a client side addition only. HMS server version is already included
We also build Impala with Hive 3.1.3. I guess if that compiles we're fine, but https://github.com/apache/hive/blob/rel/release-3.1.3/standalone-metastore/src/main/thrift/hive_metastore.thrift doesn't mention WriteNotificationLogBatchRequest.



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 May 2025 18:49:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Yida Wu, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/22869

to look at the new patch set (#3).

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................

IMPALA-14051: Use batch insert HMS API to fire insert events

Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch() available (via
HIVE-25205) starting from Hive 4. Right now, Impala is using a single
call for each partition, so firing insert statements on a huge
partitioned table can become a performance bottleneck.

This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.

Note: This optimization is applicable only for transactional partitioned
tables. The external partitioned table case is already handled.

Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/3
-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468:       hiveClient.addWriteNotificationLogInBatch(rqst);
> Was this a client-side only addition? Or does it change the required HMS se
This is a client side addition only. HMS server version is already included in impala long ago. We just have to make a change in impala to use this HMS API.


http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@470
PS1, Line 470:         String msg = "Firing write notification log request for table " + dbName + "." + tableName
> line too long (98 > 90)
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 17 May 2025 19:50:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................

IMPALA-14051: Use batch insert HMS API to fire insert events

Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch() available (via
HIVE-25205) starting from Hive 4. Right now, Impala is using a single
call for each partition, so firing insert statements on a huge
partitioned table can become a performance bottleneck.

This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.

Note: This optimization is applicable only for transactional partitioned
tables. The external partitioned table case is already handled.

Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Reviewed-on: http://gerrit.cloudera.org:8080/22869
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Riza Suminto: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11967/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 21:55:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> I'm sure we have ACID test that relevant to this change. Can you find one a
test_events_custom_configs.py#test_event_batching is the right test. But let me run the CI tests to verify that all the tests pass.



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 17:59:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG@10
PS2, Line 10: HMS API addWriteNotificationLogInBatch() 
> "HMS API addWriteNotificationLogInBatch() available starting from Hive 4."
Ack


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@440
PS2, Line 440: Fires a listener event of the type ACID_WRITE on a transactional table in metastore.
> Is this improvement limited to Hive ACID table? If yes, please mention it i
Ack


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@457
PS2, Line 457: req 
> nit: "req" is better.
Ack


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
> The partition value is added without any proper delimiter, curious what it 
Consider the table has a partition spec: (String year, int id)
It would look something like this: (["2024", 1],["2025", 1],["2026",1])


http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@466
PS2, Line 466: batc
> nit: "batchedReq" is better.
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 05:08:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 1:

(1 comment)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@470
PS1, Line 470:         String msg = "Firing write notification log request for table " + dbName + "." + tableName
line too long (98 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:20:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/22869

to look at the new patch set (#2).

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................

IMPALA-14051: Use batch insert HMS API to fire insert events

Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch(). Right now,
Impala is using a single call for each partition, so firing insert
statements on a huge partitioned table can become a performance
bottleneck.

This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.

Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/2
-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19237/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 17 May 2025 20:17:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468:       hiveClient.addWriteNotificationLogInBatch(rqst);
> This MetastoreShim is used by hive 4. MetastoreShim used by hive-3 doesn't 
Ah, the naming threw me off a bit.



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 00:00:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
> Consider the table has a partition spec: (String year, int id)
Ack. Looking at the hive code in InsertEventRequestData.java, it seems that getPartitionVal() returns a list of strings. So the printed logging could end up like "on partitions [p1, p2][p3, p4][p5, p6]". Given that, it seems okay without an additional delimiter, the brackets help separate each partition val



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 06:11:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )

Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
I'm sure we have ACID test that relevant to this change. Can you find one and try run that test with this changes, and confirm it still pass?



-- 
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 15:03:58 +0000
Gerrit-HasComments: Yes