You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2025/02/26 21:26:11 UTC
[Impala-ASF-CR] [WIP] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22554
Change subject: [WIP] IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
[WIP] IMPALA-13593: Enable event processor to consume
ALTER_PARTITIONS events from metastore
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
6 files changed, 218 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/1
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 12: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12221/
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 03:13:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#14).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 310 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/14
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 20: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 20
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 23:42:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 18:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20148/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 18
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Aug 2025 19:55:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#6).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
9 files changed, 237 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/6
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#12).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
9 files changed, 285 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/12
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20082/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 18:51:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 13:
Can the team review this patch again?
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 18:09:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 23:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20264/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 23
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 00:28:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12176/
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 19:32:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1414
PS22, Line 1414: if (!notSkipped) {
: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc();
: }
> This looks wrong - we are increasing EVENTS_SKIPPED_METRIC when the even is
reloadTableFromCatalog() returns true if the table is reloaded successfully, so the variable 'notSkipped' will be true and the if condition will not be executed. So we don't increment the metrics when the event is not skipped.
Also, the same naming convention is followed everywhere else in the file.
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2835
PS22, Line 2835: partitionsAfter_.isEmpty()
> Why would HMS send events like this? Probably this case could log a warning
I'll modify the log from INFO to WARNING.
https://issues.apache.org/jira/browse/IMPALA-12856 is one example where HMS can emit empty partition values.
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2856
PS22, Line 2856: return new SelfEventContext(dbName_, tblName_,
: Arrays.asList(getTPartitionSpecFromHmsPartition(msTbl_,
: partitionsAfter_.get(0))), partitionsAfter_.get(0).getParameters());
> An ALTER_PARTITIONS event will never merge events from multiple operations,
HMS will not merge events from HIVE and IMPALA into one ALTER_PARTITIONS event.
Whenever the alter_partitions() API is called, all the partitions of that call will be merged into one ALTER_PARTITIONS event. So the one ALTER_PARTITIONS event will either belong to HIVE or Impala(exclusively), but not both merged into one.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 00:00:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1414
PS22, Line 1414: if (!notSkipped) {
: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc();
: }
This looks wrong - we are increasing EVENTS_SKIPPED_METRIC when the even is not skipped?
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2835
PS22, Line 2835: partitionsAfter_.isEmpty()
Why would HMS send events like this? Probably this case could log a warning
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2856
PS22, Line 2856: return new SelfEventContext(dbName_, tblName_,
: Arrays.asList(getTPartitionSpecFromHmsPartition(msTbl_,
: partitionsAfter_.get(0))), partitionsAfter_.get(0).getParameters());
An ALTER_PARTITIONS event will never merge events from multiple operations, right? My concern is that merging happens, e.g. an Impala and Hive DDL are merged as event, them the "self eventness" of different partitions would not be the same.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 07:55:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#15).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 310 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/15
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh@243
PS15, Line 243: export CDP_BUILD_NUMBER=67986116
It looks like we should be good to use CDP_BUILD_NUMBER=66846208 now (7.3.1.500-30).
Can you please try?
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 16:50:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 20:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1135
PS20, Line 1135: public static class AlterPartitionsInfo {
Shouldn't this be part of Hive3MetastoreShimBase so it's available to MetastoreEvents.java no matter which MetastoreShim.java is loaded?
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 20
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 18:43:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20144/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 17
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Aug 2025 01:44:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 19: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12315/
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 19
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Aug 2025 22:00:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 3:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/22554/3/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22554/3/bin/impala-config.sh@239
PS3, Line 239: 7.3.1.200-2
Can we use a newer version like 7.3.1.200-90? I recently hit the bug fixed by HIVE-28545 and hope that we can fix it with this version bump. A newer version can avoid hitting other bugs.
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@51
PS3, Line 51: import org.apache.hadoop.hive.metastore.messaging.AlterPartitionsMessage;
Is this a problem when building on Apache Hive 3? We probably need to wrap the usage on it into a new method in
fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2888
PS3, Line 2888: ALTER_PARTITIONS EVENT
nit: use getEventDesc() to also show the event id.
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/test/resources/hive-site.xml.py@98
PS3, Line 98:
nit: use 2 spaces indention
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1671
PS3, Line 1671: event_found = True
nit: add an else-clause to log the event to ease debug in the future.
else:
logging.info("Found " + str(event))
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1676
PS3, Line 1676: # self.client.execute(":event_processor('pause')")
stale comment
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1679
PS3, Line 1679: # self.client.execute(":event_processor('start')")
stale comment
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1686
PS3, Line 1686: event_found = True
nit: Also add an else-clause here to log the event
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 May 2025 09:50:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19337/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 May 2025 00:32:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 2:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@10
PS2, Line 10: HIVE-27746 introduced ALTER_PARTITIONS event type which is an
> Please mention that we update our component versions to pick up this patch.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@12
PS2, Line 12: Impala
> Nit: in Impala.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@24
PS2, Line 24: observed while looping the test several times.
> Is this general flakiness we see in other tickets? Why would implementing t
I added bigger timeout specific to this test. This test is failing 2-3 times out of 100 because events were reaching slow to the EP, may be it has to do with my local env starting hive execution resources. I'll make it clear that I made timeouts bigger for this specific test.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@131
PS2, Line 131: ALTER_BATCH_PARTITIONS("ALTER_BATCH_PARTITIONS"),
> What is this? How is it related to the existing ALTER_PARTITIONS?
I want to use this for batch events and ALTER_PARTITIONS for the new event.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@133
PS2, Line 133: ALTER_PARTITIONS("ALTER_PARTITIONS"),
> This was already declared but unused?
Done
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2767
PS2, Line 2767: partitionsKeyVals_ =
> This never seems to be used.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2769
PS2, Line 2769: partitionsIterator_ = Preconditions.checkNotNull(
> Why is this a class property when we iterate through it immediately after a
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2788
PS2, Line 2788: infoLog("Not processing the alter partitions event {} as empty partitions are " +
> nit: "empty partitions" is ambiguous, it could be that there were no partit
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2793
PS2, Line 2793: event
> Shouldn't we include the event id also here?
Event ID is embedded in the log message by default at L#837. Self-events are common occurrences in Log, so we try to avoid duplicate info.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2797
PS2, Line 2797: if (isOlderEvent(partitionsAfter_.get(0))) {
> Could later partitions possibly be newer than the current state? That seems
All the partitions in ALTER_PARTITIONS event would have the same state, so it is enough to check one partition so see if they are older or newer events.
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py@1514
PS2, Line 1514: .format(transact_tbl, self._get_transactional_tblproperties(True)))
> There's different logic based on getHMSEventIncrementalRefreshTransactional
I want to cover the transactional and external tables scenario for the tests.
I didn't understand why we wanted to split the test into two different methods.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 May 2025 20:06:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12176/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 18:01:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jul 2025 21:22:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 16:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22554/16/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/16/tests/custom_cluster/test_events_custom_configs.py@1662
PS16, Line 1662: d
flake8: E306 expected 1 blank line before a nested definition, found 0
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 22:58:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20141/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 23:22:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 19:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12315/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 19
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Aug 2025 16:37:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#22).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 280 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/22
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 21:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1135
PS20, Line 1135: public static class AlterPartitionsInfo {
> I think it is not an ideal strategy because AlterPartitionsMessage is not a
I don't see that referenced in this class though.
What probably happens now is https://gerrit.cloudera.org/c/22554/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#2827 catches the class loading exception when it can't find AlterPartitionsInfo.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 21
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 22:39:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Reviewed-on: http://gerrit.cloudera.org:8080/22554
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 275 insertions(+), 52 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Csaba Ringhofer: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 26
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#23).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 280 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/23
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 23
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18665/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Mar 2025 18:08:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@855
PS22, Line 855: List<org.apache.hadoop.hive.metastore.api.Partition> partitions,
line too long (95 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 23:21:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 23:45:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 13:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/13/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/fe/src/test/resources/hive-site.xml.py@98
PS13, Line 98: elif VARIANT == 'events_config_change':
: # HMS config change needed for HIVE-27746 to emit ALTER_PARTITIONS event
: CONFIG.update({
: 'hive.metastore.alterPartitions.notification.v2.enabled': 'true',
: })
With CDP_BUILD_NUMBER bumped up, why this need its own VARIANT? Isn't it better to set it by default?
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/custom_cluster/test_events_custom_configs.py@1669
PS13, Line 1669: # Case-I: compute stats
# Case-I: compute stats from hive
nit: Please add blank line between test cases.
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py
File tests/util/event_processor_utils.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py@89
PS13, Line 89: timeout=10000
Is shorter default timeout (ie., 60 seconds) sufficient?
The new test_alter_partitions_event_from_metastore also set 10s timeout everywhere.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 21:11:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: [WIP] IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 1:
(7 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22554/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2748
PS1, Line 2748: // the list of partition objects of alter operation, as parsed from the NotificationEvent
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/22554/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2759
PS1, Line 2759: Preconditions.checkState(getEventType().equals(MetastoreEventType.ALTER_PARTITIONS));
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/22554/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2830
PS1, Line 2830: "processing cannot continue. Issue an invalidate command to reset " +
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/22554/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2859
PS1, Line 2859: Arrays.asList(getTPartitionSpecFromHmsPartition(msTbl_, partitionsAfter_.get(0))),
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22554/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/1/tests/custom_cluster/test_events_custom_configs.py@1523
PS1, Line 1523:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/22554/1/tests/custom_cluster/test_events_custom_configs.py@1535
PS1, Line 1535: =
flake8: E712 comparison to True should be 'if cond is True:' or 'if cond:'
http://gerrit.cloudera.org:8080/#/c/22554/1/tests/custom_cluster/test_events_custom_configs.py@1545
PS1, Line 1545: =
flake8: E712 comparison to True should be 'if cond is True:' or 'if cond:'
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 21:29:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19607/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Jun 2025 04:28:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12150/
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jul 2025 07:02:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG@19
PS11, Line 19: . This can cause partitions to be refreshed even on
: trivial changes to them.
> This sounds like a potential perf regression to me. Couldn't HMS add the be
Agree. But HIVE-27746 is controlled with a config, and it is not turned on by default. So we are not making things bad at this moment. The plus side of the patch is we don't have to batch partition events together in the event processor and reload them in one-go. I plan to make improvements in HMS (either adding before-partitions or marking which partitions really changed in the notification event).
I also plan to do non-functional qe testing to understand the perf regression after merging this patch to understand the advantages of having fewer events for large partitioned tables.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jul 2025 00:11:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG@19
PS11, Line 19: . This can cause partitions to be refreshed even on
: trivial changes to them.
This sounds like a potential perf regression to me. Couldn't HMS add the before_partitions or couldn't Impala use its own cached partition object to check if the change is trivial?
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jul 2025 17:42:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 7:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/22554/7/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/22554/7/be/src/catalog/catalog-server.cc@223
PS7, Line 223: ALTER_PARTITIONS
Is it intended to skip ALTER_PARTITION events after this patch? Are there scenarios that HMS still generate ALTER_PARTITION events, e.g. when the alter_partition HMS RPC is called?
I think it'd be safer to keep both types.
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@634
PS7, Line 634: updatedFields.put("table", msTbl);
: updatedFields.put("partitions", partitionsAfter);
: updatedFields.put("isTruncate", isTruncateOp);
nit: It might be better to define a class for this. Then we don't need cast in the call site.
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2860
PS7, Line 2860: "ALTER_PARTITIONS event"
nit: use getEventDesc()
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2866
PS7, Line 2866: }
nit: L2843-2866 are quite similar to codes in AlterPartitionEvent#processTableEvent(). Can we extract them into a method to simplify the code?
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2878
PS7, Line 2878: boolean incrementalRefresh =
: BackendConfig.INSTANCE.getHMSEventIncrementalRefreshTransactionalTable();
: if (incrementalRefresh) {
: reloadPartitionsFromEvent(partitionsAfter_, getEventDesc()
: + " FOR TRANSACTIONAL TABLE");
: } else {
: boolean notSkipped = reloadTableFromCatalog(true);
: if (!notSkipped) {
: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc();
: }
: }
nit: These are also simlar to the implementation in AlterPartitionEvent. Can we extract them into a method to share them?
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1671
PS3, Line 1671: EventProcessorUtils.wait_for_event_processing(self, 10)
> Ack
not done yet.
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1676
PS3, Line 1676: # Case-II: compute stats from impala
> Ack
not done yet.
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1679
PS3, Line 1679: events_skipped_before = EventProcessorUtils.get_int_metric('events-skipped', 0)
> Ack
not done yet.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 13:27:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@12
PS2, Line 12: Impala
Nit: in Impala.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2793
PS2, Line 2793: event
Shouldn't we include the event id also here?
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py@1557
PS2, Line 1557: #
Should this be uncommented? Or does it belong to Case-IV?
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 22 Apr 2025 17:07:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#5).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
9 files changed, 238 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/5
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19324/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 May 2025 22:07:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19788/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jul 2025 06:10:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#21).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 309 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/21
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 21
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1135
PS20, Line 1135: * @param event Commit transaction event
> I don't see that referenced in this class though.
I have addressed it in my latest patchset. Thanks for the suggestion.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 23:19:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 14: Code-Review+1
(2 comments)
Please ask follow up review from either Quanlong or Csaba.
http://gerrit.cloudera.org:8080/#/c/22554/14/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/14/tests/custom_cluster/test_events_custom_configs.py@1670
PS14, Line 1670: compute stats
nit: compute stats from hive
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py
File tests/util/event_processor_utils.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py@89
PS13, Line 89: timeout=10, e
> My bad!! This was committed by mistake. I increased this timeout for debugg
Done
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 18:01:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#18).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 309 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/18
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 18
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 25: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 25
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 06:53:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 21:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20228/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 21
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 22:07:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/22554/3/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22554/3/bin/impala-config.sh@239
PS3, Line 239: 7.3.1.500-3
> Can we use a newer version like 7.3.1.200-90? I recently hit the bug fixed
Ack
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@51
PS3, Line 51: import org.apache.hadoop.hive.metastore.messaging.AlterPartitionsMessage;
> Is this a problem when building on Apache Hive 3? We probably need to wrap
I'm not introducing new APIs, we hit the same old alter_partitions() API but the internal working is tweaked to fire ALTER_PARTITIONS event based on a config in metastore. So it'll not be a problem with HIVE 3 if we are using the bin/impala-config.sh present in this patch.
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2888
PS3, Line 2888: " FOR TRANSACTIONAL T
> nit: use getEventDesc() to also show the event id.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1671
PS3, Line 1671: event_found = True
> nit: add an else-clause to log the event to ease debug in the future.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1676
PS3, Line 1676: # self.client.execute(":event_processor('pause')")
> stale comment
Ack
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1679
PS3, Line 1679: # self.client.execute(":event_processor('start')")
> stale comment
Ack
http://gerrit.cloudera.org:8080/#/c/22554/3/tests/custom_cluster/test_events_custom_configs.py@1686
PS3, Line 1686: event_found = True
> nit: Also add an else-clause here to log the event
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 May 2025 21:55:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@51
PS3, Line 51: import org.apache.hadoop.hive.metastore.messaging.CreateTableMessage;
> Does this class (AlterPartitionsMessage) exist in Apache Hive 3.1.3? I don'
I have an incorrect understanding about the build jars. Addressed your concern in the latest patchset
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 May 2025 00:23:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#16).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 307 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/16
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 18:
Please rebase to pick up IMPALA-14308.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 18
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Aug 2025 21:16:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12221/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jul 2025 21:05:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 04:25:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/13/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/fe/src/test/resources/hive-site.xml.py@98
PS13, Line 98: elif VARIANT == 'events_config_change':
: # HMS config change needed for HIVE-27746 to emit ALTER_PARTITIONS event
: CONFIG.update({
: 'hive.metastore.alterPartitions.notification.v2.enabled': 'true',
: })
> With CDP_BUILD_NUMBER bumped up, why this need its own VARIANT? Isn't it be
This is not a default in the hive metastore yet. Other listeners are not ready to consume this event yet.
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/custom_cluster/test_events_custom_configs.py@1669
PS13, Line 1669:
> # Case-I: compute stats from hive
Ack
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py
File tests/util/event_processor_utils.py:
http://gerrit.cloudera.org:8080/#/c/22554/13/tests/util/event_processor_utils.py@89
PS13, Line 89: timeout=10, e
> Is shorter default timeout (ie., 60 seconds) sufficient?
My bad!! This was committed by mistake. I increased this timeout for debugging event procesor in my local env.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 17:32:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 25:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12391/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 25
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Aug 2025 23:25:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 23: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1414
PS22, Line 1414: if (!notSkipped) {
: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc();
: }
> reloadTableFromCatalog() returns true if the table is reloaded successfully
Oops, you are right, I was somehow confused by the double negation.
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2835
PS22, Line 2835: partitionsAfter_.isEmpty()
> I'll modify the log from INFO to WARNING.
ack
http://gerrit.cloudera.org:8080/#/c/22554/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2856
PS22, Line 2856: return new SelfEventContext(dbName_, tblName_,
: Arrays.asList(getTPartitionSpecFromHmsPartition(msTbl_,
: partitionsAfter_.get(0))), partitionsAfter_.get(0).getParameters());
> HMS will not merge events from HIVE and IMPALA into one ALTER_PARTITIONS ev
ack
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 23
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Aug 2025 12:53:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 2:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@10
PS2, Line 10: HIVE-27746 introduced ALTER_PARTITIONS event type which is an
Please mention that we update our component versions to pick up this patch.
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@24
PS2, Line 24: observed while looping the test several times.
Is this general flakiness we see in other tickets? Why would implementing this potentially make event processing more variable? Or do you think they're related to the Hive version update?
If it's just too slow for this particular test, we should specify the timeout for this test rather than changing the default.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@131
PS2, Line 131: ALTER_BATCH_PARTITIONS("ALTER_BATCH_PARTITIONS"),
What is this? How is it related to the existing ALTER_PARTITIONS?
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@133
PS2, Line 133: ALTER_PARTITIONS("ALTER_PARTITIONS"),
This was already declared but unused?
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2767
PS2, Line 2767: partitionsKeyVals_ =
This never seems to be used.
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2769
PS2, Line 2769: partitionsIterator_ = Preconditions.checkNotNull(
Why is this a class property when we iterate through it immediately after and don't use it outside the constructor?
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2788
PS2, Line 2788: infoLog("Not processing the alter partitions event {} as empty partitions are " +
nit: "empty partitions" is ambiguous, it could be that there were no partitions or that each partition was empty. Maybe say "no partitions".
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2797
PS2, Line 2797: if (isOlderEvent(partitionsAfter_.get(0))) {
Could later partitions possibly be newer than the current state? That seems like it would be an error on HMS's end to send such a message.
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py@1514
PS2, Line 1514: .format(transact_tbl, self._get_transactional_tblproperties(True)))
There's different logic based on getHMSEventIncrementalRefreshTransactionalTable. Is that what this table is covering?
This could be split into two test methods, using class-level CustomClusterTestSuite.with_args config to start a single cluster for both tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 16 Apr 2025 23:35:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22554/7/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/22554/7/be/src/catalog/catalog-server.cc@223
PS7, Line 223: NS,CREATE_TABLE,
> Is it intended to skip ALTER_PARTITION events after this patch? Are there s
I intend to remove ADD_PARTITION,ALTER_PARTITION, DROP_PARTITION which are duplicates and add ALTER_PARTITIONS
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@634
PS7, Line 634: tableReloadInfo = new TableReloadInfo(msTbl, partitionsAfter, isTruncateOp);
: } catch (Exception e) {
: throw new MetastoreNotificationException(e);
> nit: It might be better to define a class for this. Then we don't need cast
Ack
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2860
PS7, Line 2860: ed from a single initial
> nit: use getEventDesc()
Ack
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2866
PS7, Line 2866: MetastoreTableEvent {
> nit: L2843-2866 are quite similar to codes in AlterPartitionEvent#processTa
Ack
http://gerrit.cloudera.org:8080/#/c/22554/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2878
PS7, Line 2878:
: @Override
: public MetastoreEvent addToBatchEvents(MetastoreEvent event) {
: Preconditions.checkState(canBeBatched(event));
: batchedEvents_.add((T) event);
: return this;
: }
:
: @Override
: public int getNumberOfEvents() { return batchedEvents_.size(); }
:
> nit: These are also simlar to the implementation in AlterPartitionEvent. Ca
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jul 2025 05:52:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#8).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
9 files changed, 286 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/8
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12150/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jul 2025 05:31:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 3: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/2//COMMIT_MSG@24
PS2, Line 24: event. Also, bigger time outs were added in this test as there was
> I added bigger timeout specific to this test. This test is failing 2-3 time
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@131
PS2, Line 131: ALTER_DATABASE("ALTER_DATABASE"),
> I want to use this for batch events and ALTER_PARTITIONS for the new event.
Ack
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/2/tests/custom_cluster/test_events_custom_configs.py@1514
PS2, Line 1514: # Test creating table with the name after rename it to another
> I want to cover the transactional and external tables scenario for the test
Slightly simpler to identify if a single case fails. Not a significant concern.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 May 2025 23:48:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#3).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 197 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/3
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/22554/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@51
PS3, Line 51: import org.apache.hadoop.hive.metastore.messaging.AlterPartitionsMessage;
> I'm not introducing new APIs, we hit the same old alter_partitions() API bu
Does this class (AlterPartitionsMessage) exist in Apache Hive 3.1.3? I don't see it here:
https://github.com/apache/hive/tree/rel/release-3.1.3/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 May 2025 23:24:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: [WIP] IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 21:53:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#24).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 275 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/24
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 24
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 25: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 25
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 04:34:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 11: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12183/
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jul 2025 19:08:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20080/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 17:58:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 21:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/20/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1135
PS20, Line 1135: public static class AlterPartitionsInfo {
> Shouldn't this be part of Hive3MetastoreShimBase so it's available to Metas
I think it is not an ideal strategy because AlterPartitionsMessage is not available in the Apache Hive 3 shim, which would result in a ClassNotFoundException at runtime.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 21
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 22:31:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#2).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. It would be a good optimization to include this Impala so that
the number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in test as there was flakiness
observed while looping the test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
6 files changed, 198 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/2
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19139/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 May 2025 20:33:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#4).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
7 files changed, 201 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/4
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 19:54:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12242/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 22:05:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#13).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
9 files changed, 307 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/13
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 15:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG@19
PS11, Line 19: . This can cause partitions to be refreshed even on
: trivial changes to them.
> Agree. But HIVE-27746 is controlled with a config, and it is not turned on
Thanks for the explanation?
Can you create a ticket for this issue with HIVE-27746 and mention it here?
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@476
PS15, Line 476: getPartitions
getPartitionsAfter()?
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@611
PS15, Line 611: returns them in a map.
this should be updated
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1132
PS15, Line 1132: TableReloadInfo
I don't get the naming - this class seems specific to ALTER_PARTITIONS, but this is not reflected in the name. Or this could be used in other types too in the future?
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1150
PS15, Line 1150: getMsTable
Maybe getMsTableAfter would be more exact?
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1677
PS15, Line 1677: assert batch_events_after == batch_events_before # verify there are no new batches
Can we also verify to the total number of events generated by ANALYZE TABLE?
Also, should this case generate an ALTER_PARTITIONS event? We could verify it similarly to other cases.
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1704
PS15, Line 1704: event_found = False
: for event in events:
: if event.eventType == "ALTER_PARTITIONS":
: event_found = True
: else:
: logging.debug("Found event: " + str(event))
: assert event_found
The same is done twice, there could be a function for it
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 16:23:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 17:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22554/17/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/17/tests/custom_cluster/test_events_custom_configs.py@1671
PS17, Line 1671:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/22554/17/tests/custom_cluster/test_events_custom_configs.py@1671
PS17, Line 1671:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 17
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Aug 2025 01:21:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 16:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG@19
PS11, Line 19: . This can cause partitions to be refreshed even on
: trivial changes to them.
> Thanks for the explanation?
Created https://issues.apache.org/jira/browse/HIVE-29141 to track this issue.
http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh@243
PS15, Line 243: export CDP_BUILD_NUMBER=66846208
> It looks like we should be good to use CDP_BUILD_NUMBER=66846208 now (7.3.1
Ack
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@476
PS15, Line 476: getPartitions
> getPartitionsAfter()?
This method is for TableReload class, it make sense to keep getPartitions() as the name of the method: https://gerrit.cloudera.org/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#1158
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@611
PS15, Line 611: returns them in a Alte
> this should be updated
Ack
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1132
PS15, Line 1132:
> I don't get the naming - this class seems specific to ALTER_PARTITIONS, but
I initially wanted to use to for ReloadEvent as well but it seems it is not useful as the fields are different. I'll rename the class appropriately.
http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1150
PS15, Line 1150:
> Maybe getMsTableAfter would be more exact?
There'll be only one table object for the alter partition event. That's the naming convention we have followed in other places.
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1677
PS15, Line 1677:
> Can we also verify to the total number of events generated by ANALYZE TABLE
The number of events for managed and external tables varies. Also, it might cause some flakiness in the future if some transactional events don't make it in time. I think verifying no batches were formed and the generation of ALTER_PARTITIONS event is good enough for this test.
Yeah, analyze table compute statistics will call alter-table api in hive metastore, which generates alter partitions events as they can update stats in metastore.
http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1704
PS15, Line 1704: events = EventProcessorUtils.get_next_notification(self.hive_client, last_event_id)
: events_skipped_after = EventProcessorUtils.get_int_metric('events-skipped', 0)
: assert _verify_alter_partitions_event(events)
: assert events_skipped_after > events_skipped_before
:
: # Case-IV: Truncate table from Hive is currently generating single alter_partition
: # events. HIVE-286
> The same is done twice, there could be a function for it
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 22:56:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Michael Smith, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22554
to look at the new patch set (#17).
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
---
M be/src/catalog/catalog-server.cc
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 309 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/22554/17
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 17
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 20:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12351/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 20
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 18:32:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12183/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jul 2025 17:24:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 22: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 22
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 23:28:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13593: Enable event processor to consume ALTER PARTITIONS events from metastore
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore
......................................................................
Patch Set 24:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20291/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22554
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Gerrit-Change-Number: 22554
Gerrit-PatchSet: 24
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Aug 2025 23:51:40 +0000
Gerrit-HasComments: No