You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2025/05/08 21:18:47 UTC
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22869
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
IMPALA-14051: Use batch insert HMS API to fire insert events
Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch(). Right now,
Impala is using a single call for each partition, so firing insert
statements on a huge partitioned table can become a performance
bottleneck.
This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.
Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 26 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/1
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468: hiveClient.addWriteNotificationLogInBatch(rqst);
Was this a client-side only addition? Or does it change the required HMS server version Impala works with?
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:27:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 03:59:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:44:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG@10
PS2, Line 10: HMS API addWriteNotificationLogInBatch().
"HMS API addWriteNotificationLogInBatch() available starting from Hive 4."
If you can also mention the Hive JIRA number, that will be great.
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@440
PS2, Line 440: Fires a listener event of the type ACID_WRITE on a transactional table in metastore.
Is this improvement limited to Hive ACID table? If yes, please mention it in commit message.
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@457
PS2, Line 457: rqst
nit: "req" is better.
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@466
PS2, Line 466: rqst
nit: "batchedReq" is better.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Jun 2025 22:44:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3: Code-Review+2
(1 comment)
Carry Michael's +1.
Please monitor downstream exhaustive tests for unforeseen cases.
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> Tests ran green. Are we good to merge this?
Done
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:08:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
(1 comment)
@Riza
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> test_events_custom_configs.py#test_event_batching is the right test. But le
Tests ran green. Are we good to merge this?
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 22:04:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
The partition value is added without any proper delimiter, curious what it would look like in the later debug logging
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 02 Jun 2025 23:03:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19370/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 05:35:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468: hiveClient.addWriteNotificationLogInBatch(rqst);
> We also build Impala with Hive 3.1.3. I guess if that compiles we're fine,
This MetastoreShim is used by hive 4. MetastoreShim used by hive-3 doesn't support this method: https://github.com/apache/impala/blob/master/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L385-L386
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 May 2025 23:57:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468: hiveClient.addWriteNotificationLogInBatch(rqst);
> This is a client side addition only. HMS server version is already included
We also build Impala with Hive 3.1.3. I guess if that compiles we're fine, but https://github.com/apache/hive/blob/rel/release-3.1.3/standalone-metastore/src/main/thrift/hive_metastore.thrift doesn't mention WriteNotificationLogBatchRequest.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 May 2025 18:49:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Yida Wu, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22869
to look at the new patch set (#3).
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
IMPALA-14051: Use batch insert HMS API to fire insert events
Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch() available (via
HIVE-25205) starting from Hive 4. Right now, Impala is using a single
call for each partition, so firing insert statements on a huge
partitioned table can become a performance bottleneck.
This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.
Note: This optimization is applicable only for transactional partitioned
tables. The external partitioned table case is already handled.
Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/3
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468: hiveClient.addWriteNotificationLogInBatch(rqst);
> Was this a client-side only addition? Or does it change the required HMS se
This is a client side addition only. HMS server version is already included in impala long ago. We just have to make a change in impala to use this HMS API.
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@470
PS1, Line 470: String msg = "Firing write notification log request for table " + dbName + "." + tableName
> line too long (98 > 90)
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 17 May 2025 19:50:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
IMPALA-14051: Use batch insert HMS API to fire insert events
Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch() available (via
HIVE-25205) starting from Hive 4. Right now, Impala is using a single
call for each partition, so firing insert statements on a huge
partitioned table can become a performance bottleneck.
This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.
Note: This optimization is applicable only for transactional partitioned
tables. The external partitioned table case is already handled.
Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Reviewed-on: http://gerrit.cloudera.org:8080/22869
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Riza Suminto: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11967/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 21:55:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
> I'm sure we have ACID test that relevant to this change. Can you find one a
test_events_custom_configs.py#test_event_batching is the right test. But let me run the CI tests to verify that all the tests pass.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jun 2025 17:59:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/2//COMMIT_MSG@10
PS2, Line 10: HMS API addWriteNotificationLogInBatch()
> "HMS API addWriteNotificationLogInBatch() available starting from Hive 4."
Ack
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@440
PS2, Line 440: Fires a listener event of the type ACID_WRITE on a transactional table in metastore.
> Is this improvement limited to Hive ACID table? If yes, please mention it i
Ack
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@457
PS2, Line 457: req
> nit: "req" is better.
Ack
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
> The partition value is added without any proper delimiter, curious what it
Consider the table has a partition spec: (String year, int id)
It would look something like this: (["2024", 1],["2025", 1],["2026",1])
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@466
PS2, Line 466: batc
> nit: "batchedReq" is better.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 05:08:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 1:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@470
PS1, Line 470: String msg = "Firing write notification log request for table " + dbName + "." + tableName
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 21:20:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22869
to look at the new patch set (#2).
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
IMPALA-14051: Use batch insert HMS API to fire insert events
Firing insert statements from Impala can be improved by leveraging the
batch insert HMS API addWriteNotificationLogInBatch(). Right now,
Impala is using a single call for each partition, so firing insert
statements on a huge partitioned table can become a performance
bottleneck.
This patch addresses the above concern by leveraging
addWriteNotificationLogInBatch() API to fire batch insert partition
events.
Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
2 files changed, 27 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/22869/2
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19237/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 17 May 2025 20:17:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@468
PS1, Line 468: hiveClient.addWriteNotificationLogInBatch(rqst);
> This MetastoreShim is used by hive 4. MetastoreShim used by hive-3 doesn't
Ah, the naming threw me off a bit.
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 00:00:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22869/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@462
PS2, Line 462: sb.append(insertData.getPartitionVal());
> Consider the table has a partition spec: (String year, int id)
Ack. Looking at the hive code in InsertEventRequestData.java, it seems that getPartitionVal() returns a list of strings. So the printed logging could end up like "on partitions [p1, p2][p3, p4][p5, p6]". Given that, it seems okay without an additional delimiter, the brackets help separate each partition val
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 06:11:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14051: Use batch insert HMS API to fire insert events
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22869 )
Change subject: IMPALA-14051: Use batch insert HMS API to fire insert events
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22869/3//COMMIT_MSG@22
PS3, Line 22: Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
I'm sure we have ACID test that relevant to this change. Can you find one and try run that test with this changes, and confirm it still pass?
--
To view, visit http://gerrit.cloudera.org:8080/22869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff4a2691631fe9e9e1dc07714c69580a4ace6a8b
Gerrit-Change-Number: 22869
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 03 Jun 2025 15:03:58 +0000
Gerrit-HasComments: Yes