You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2026/09/29 22:15:26 UTC
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
k.venureddy2103@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24975
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread is scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when an event is enqueued to one of
its DbProcessors, and when all the TableProcessors waiting on a
DbBarrierEvent have reached it.
- TableEventExecutor is signalled when an event is enqueued to one
of its TableProcessors, and when a DbBarrierEvent its
TableProcessor waits on is processed.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
An executor still wakes up on a timeout in the following cases:
- TableEventExecutor retries every 10 ms while it has outstanding
events that could not be processed yet, e.g. events waiting on a
RenameTableBarrierEvent.
- DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle
for min_event_processor_idle_ms.
Otherwise, the executor waits indefinitely until it is signalled.
Since events are processed as soon as they are enqueued, instead of
waiting up to 10 ms for the next tick, this also removes the
scheduling latency from event processing.
Testing:
- Ran existing tests.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
7 files changed, 228 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 13:23:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14175/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 08:43:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py
File tests/custom_cluster/test_event_processing_perf.py:
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py@82
PS4, Line 82: "--enable_hierarchical_event_processing=false")
Do we need to disable hierarchical event processing in these tests? If so, can we mention it in the commit message?
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 07:23:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java@807
PS4, Line 807: for (Map.Entry<String, DbProcessor> entry : dbProcessors_.entrySet()) {
nit: we don't use the keys here so we can iterate dbProcessors_.values() directly.
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
File fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java@85
PS4, Line 85: return false;
Should we add a warning log for this?
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java@88
PS4, Line 88: the wait has elapsed
The javadoc of Object.wait() mentions that
> A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup. While this will rarely occur in practice, applications must guard against it by testing for the condition that should have caused the thread to be awakened, and continuing to wait if the condition is not satisfied.
Maybe update this comment and the javadoc of this method to mention that a positive timeout is a best-effort upper bound (callers must tolerate early returns).
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
File fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@113
PS4, Line 113:
nit: remove blank line and empty @return line
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@122
PS4, Line 122:
nit: remove blank line and empty @return line
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@129
PS4, Line 129: * @param eventType
: * @param skipped
nit: remove these or add descriptions
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@473
PS4, Line 473: if (!waiter_.await(0)) break;
Will this be a problem in the following scenario?
1. An event fails to process for table T. It stays in the corresponding TableProcessor.events_ and this executor parks on await(0).
2. User manually runs :event_processor('start') or :event_processor('start', <fromEventId>).
Result: the parked executors with their stale queued events stay blocked on await(0). If fromEventId skips past those event ids, they are never re-dispatched, so they sit unprocessed until some unrelated future event for table T happens to signal that executor.
Invoking signal() in EventExecutorService.makeActive()/start() for all db/table executors seems a fix for this.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 10 Oct 2026 09:42:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
https://gerrit.cloudera.org/24975
to look at the new patch set (#9).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
1) When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
2) When an event fails to process, the event processor changes to ERROR
or NEEDS_INVALIDATE status and the executors stop processing, leaving
the failed event and the events after it queued on the executors.
These stale events were not discarded. So upon event processor start,
they were processed first, regardless of the event id the event
processor is started from.
To fix it, the events queued on the executors are now discarded when
the event processor changes to ERROR or NEEDS_INVALIDATE. The greatest
synced event is retained, and the last dispatched event is moved back
to it. Upon start, event processing resumes from the greatest synced
event id, the latest event id or the given event id, and the events
after it are fetched from HMS again.
3) In hierarchical mode, clearing the event executors upon event
processing failure or global INVALIDATE METADATA discards the queued
events and makes their databases and tables appear processed, so HMS
event sync requests could succeed with stale metadata.
EventExecutorService now counts the number of times it is cleared. A
request captures the count before checking its databases and tables.
If the count changes or the event processor cannot process events,
the request waits for the greatest synced event id to reach its latest
event id instead, since clear() retains the greatest synced event and
the catalog reset advances it. The request fails if the event
processor goes to ERROR or NEEDS_INVALIDATE, or times out.
4) This also fixes a race in RenameTableBarrierEvent, where the pseudo
drop and create table event threads could both see both events
processed and skipped, and both increment the events-skipped metric.
Event-driven wakeups made it more likely, causing
test_rename_table_hierarchical_one_db_and_two_table_threads to fail
intermittently. Since the pseudo create table event is always
processed after the pseudo drop table event, only the create now
updates the skipped metric and the delete event log, and injects the
error for testing.
Testing:
- Ran existing tests.
- Added test_event_processor_start_upon_error to verify that the
failed event is retried upon :event_processor('start'), and is not
retried upon :event_processor('start', -1) even when a subsequent
event is enqueued for the same table. Multiple INSERT events are
fetched in a batch, so that the events queued after the failed
event are discarded upon failure.
- Added test_hms_event_sync_upon_catalog_reset to verify that a query
waiting for HMS events to be synced succeeds with the latest metadata
when a global INVALIDATE METADATA discards the events queued on the
event executors.
- The non-hierarchical variants of the tests are now set
--enable_hierarchical_event_processing=false explicitly. The flag
defaults to true, so they were running in hierarchical mode.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/EventProcessException.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_error.py
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
12 files changed, 624 insertions(+), 178 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/9
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14484/ DRY_RUN=true
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 13:41:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#3).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
Testing:
- Ran existing tests.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
10 files changed, 294 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/3
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 08:00:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#6).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
1) When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
2) When an event fails to process, the event processor changes to ERROR
or NEEDS_INVALIDATE status and the executors stop processing, leaving
the failed event and the events after it queued on the executors.
These stale events were not discarded. So upon event processor start,
they were processed first, regardless of the event id the event
processor is started from.
To fix it, the events queued on the executors are now discarded when
the event processor changes to ERROR or NEEDS_INVALIDATE. The greatest
synced event is retained, and the last dispatched event is moved back
to it. Upon start, event processing resumes from the greatest synced
event id, the latest event id or the given event id, and the events
after it are fetched from HMS again.
3) HMS event sync requests in hierarchical mode now fail if the event
processor cannot process events, or the last dispatched event moves
back below the event id they wait for. Discarded events make their
databases and tables appear processed, so the requests could
otherwise succeed with stale metadata.
4) This also fixes a race in RenameTableBarrierEvent, where the pseudo
drop and create table event threads could both see both events
processed and skipped, and both increment the events-skipped metric.
Event-driven wakeups made it more likely, causing
test_rename_table_hierarchical_one_db_and_two_table_threads to fail
intermittently. Since the pseudo create table event is always
processed after the pseudo drop table event, only the create now
updates the skipped metric and the delete event log, and injects the
error for testing.
Testing:
- Ran existing tests.
- Added test_event_processor_start_upon_error to verify that the
failed event is retried upon :event_processor('start'), and is not
retried upon :event_processor('start', -1) even when a subsequent
event is enqueued for the same table.
- The non-hierarchical variants of the tests are now set
--enable_hierarchical_event_processing=false explicitly. The flag
defaults to true, so they were running in hierarchical mode.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_error.py
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 502 insertions(+), 153 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/6
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#7).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
1) When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
2) When an event fails to process, the event processor changes to ERROR
or NEEDS_INVALIDATE status and the executors stop processing, leaving
the failed event and the events after it queued on the executors.
These stale events were not discarded. So upon event processor start,
they were processed first, regardless of the event id the event
processor is started from.
To fix it, the events queued on the executors are now discarded when
the event processor changes to ERROR or NEEDS_INVALIDATE. The greatest
synced event is retained, and the last dispatched event is moved back
to it. Upon start, event processing resumes from the greatest synced
event id, the latest event id or the given event id, and the events
after it are fetched from HMS again.
3) HMS event sync requests in hierarchical mode now fail if the event
processor cannot process events, or the event executors are cleared
while they wait, i.e. upon event processing failure or global
INVALIDATE METADATA. Clearing discards the queued events and makes
their databases and tables appear processed, so the requests could
otherwise succeed with stale metadata. EventExecutorService counts the
number of times it is cleared. A request captures the count before
checking its databases and tables, and fails if the count changes.
4) This also fixes a race in RenameTableBarrierEvent, where the pseudo
drop and create table event threads could both see both events
processed and skipped, and both increment the events-skipped metric.
Event-driven wakeups made it more likely, causing
test_rename_table_hierarchical_one_db_and_two_table_threads to fail
intermittently. Since the pseudo create table event is always
processed after the pseudo drop table event, only the create now
updates the skipped metric and the delete event log, and injects the
error for testing.
Testing:
- Ran existing tests.
- Added test_event_processor_start_upon_error to verify that the
failed event is retried upon :event_processor('start'), and is not
retried upon :event_processor('start', -1) even when a subsequent
event is enqueued for the same table. Multiple INSERT events are
fetched in a batch, so that the events queued after the failed
event are discarded upon failure.
- Added test_hms_event_sync_fails_upon_catalog_reset to verify that a
query waiting for HMS events to be synced fails when a global
INVALIDATE METADATA discards the events queued on the event
executors.
- The non-hierarchical variants of the tests are now set
--enable_hierarchical_event_processing=false explicitly. The flag
defaults to true, so they were running in hierarchical mode.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/EventProcessException.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_error.py
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
12 files changed, 587 insertions(+), 156 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/7
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14205/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 03:24:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14362/
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 31 Oct 2026 03:31:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14304/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 17:47:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24767/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 17:14:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14198/
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 16:07:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6:
(8 comments)
Thanks for the review, Quanlong. All comments are addressed in the latest patch set 5.
Also, While testing, test_rename_table_hierarchical_one_db_and_two_table_threads failed intermittently. The new wakeup signal exposed an existing race in RenameTableBarrierEvent.updateStatus(): both pseudo-events could see the other as processed and both increment the skipped metric. Now only the pseudo create table event does the final bookkeeping (skipped metric and delete event log). And injectErrorIfNeeded() runs only after the create.
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java@807
PS4, Line 807: if (!eventProcessor_.canProcessEventInCurrentStatus()) break;
> nit: we don't use the keys here so we can iterate dbProcessors_.values() di
done
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
File fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java@85
PS4, Line 85: * @return False if the executor is stopped or the waiting thread is interrupted, and
> Should we add a warning log for this?
Done
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java@88
PS4, Line 88: nized boolean await(
> The javadoc of Object.wait() mentions that
Done
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
File fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@113
PS4, Line 113: synchronized boolean isDropProcessed() {
> nit: remove blank line and empty @return line
Done
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@122
PS4, Line 122: return createProcessed_;
> nit: remove blank line and empty @return line
Done
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@129
PS4, Line 129: */
: synchronized void
> nit: remove these or add descriptions
Done. Added descriptions for the parameters.
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@473
PS4, Line 473: if (!waiter_.await(0)) break;
> Will this be a problem in the following scenario?
Thanks, good catch. I had assumed invalidate metadata was the only way to recover from ERROR/NEEDS_INVALIDATE. But :event_processor('pause') and :event_processor('start') are both accepted in these states, so the scenario is possible.
Signalling the executors in makeActive()/start() alone wouldn't be enough, though. The stale events would then be processed and could overlap with the events re-fetched from the greatest synced event. With start(-1) or start(<fromEventId>), they would even be applied after the user asked to skip them.
The same stale events also caused a related problem. :event_processor('pause') after a failure could block for up to an hour in pauseGracefully(). ensureEventsProcessedInHierarchicalMode() waits for the outstanding event count to reach 0. But the failed event and the events queued behind it never drains. Parked TableEventExecutor isn't signalled on pause, so it never retries. And even when the failed event is retried, a deterministic failure can again bring back the event processor to ERROR.
So have fixed to discard events on failure. And start resumes and fetches the events depending on the starting point(i.e., start()/start(-1)/start(<fromEventId>)). No executor is left holding queued events, so nothing stays parked with stale work, and pauseGracefully() returns immediately after a failure. And in syncMetadataWithHierarchicalEventProcessing(), HMS event sync requests now fail if the status changes while it is waiting for sync to finish.
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py
File tests/custom_cluster/test_event_processing_perf.py:
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py@82
PS4, Line 82: "--enable_hierarchical_event_processing=false")
> Do we need to disable hierarchical event processing in these tests? If so,
Done. All these tests exists with explicit "--enable_hierarchical_event_processing=true" as well right next to the corresponding tests. These tests were added before changing default value of enable_hierarchical_event_processing to true. Expected to run these tests with enable_hierarchical_event_processing false. After change of enable_hierarchical_event_processing default value to true, tests run both times with hep enabled. Updated commit message now.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 06:38:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14293/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 19 Oct 2026 06:14:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24575/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 19 Oct 2026 06:26:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 19 Oct 2026 10:48:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8:
(3 comments)
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1429
PS8, Line 1429: while this lock is held
nit: maybe more clear saying "and no events will be dispatched while this lock is held".
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@2181
PS8, Line 2181: String error = getSyncInterruptedError(latestEventId, clearCount);
Is it possible that getGreatestSyncedEventId() has reached latestEventId at some point? Maybe we can check this in getSyncInterruptedError() and rescue some requests.
https://gerrit.cloudera.org/#/c/24975/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
https://gerrit.cloudera.org/#/c/24975/8/tests/custom_cluster/test_events_custom_configs.py@2122
PS8, Line 2122: succeed with stale metadata."""
Shouldn't we let the query succeed since global INVALIDATE METADATA has cleared all the stale metadata and advance the greatest synced event id? I think that's the behavior when HEP is disabled. Maybe HEP has the same behavior before IMPALA-14535.
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 12 Nov 2026 02:49:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1428
PS6, Line 1428: // synced event retained by clear(). It is set after clear(), since the processed
> nit: It's worth mentioning here that lastSycnedEventId is the last dispatch
Done
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1429
PS6, Line 1429: // log cannot change once the executors are cleared while this lock is held. So the
> Is it possible that another TableExecutor thread updates the processedLog_
Yes. The two getters take the EventExecutorService lock separately, and executor threads can advance the greatest synced event between them and before clear(). So the id and time could come from different events, and clear() could retain a later event than the one lastSyncedEventId_ was moved back to.
So, moved the update after clear() instead. Once clear() returns, the processed log only has the retained entry, and it cannot change while updateStatusOnFailure() holds this lock. While looking at this, found that the last dispatched event id check in getSyncInterruptedError() could not reliably detect discarded events. So I replaced it with a clear count in EventExecutorService, incremented in clear() and shutdown(). The sync request captures it under this lock before its isProcessed() checks, and fails if it changes. Added test_hms_event_sync_fails_upon_catalog_reset for the reset case.
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
File fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@187
PS6, Line 187: Preconditions.checkState(state_.isDropProcessed(),
> nit: let's avoid using Preconditions checks without error messages, e.g.
Done
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@373
PS6, Line 373: throw new EventProcessException(event.getEvent(), e);
: }
: }
: }
: }
:
> nit: this appears 3 times. How about moving it to the class comment of Even
Done
http://gerrit.cloudera.org:8080/#/c/24975/6/tests/custom_cluster/test_event_processing_error.py
File tests/custom_cluster/test_event_processing_error.py:
http://gerrit.cloudera.org:8080/#/c/24975/6/tests/custom_cluster/test_event_processing_error.py@61
PS6, Line 61: # Pause the event processor so that all the INSERT events are fetched in a batch and
> Can we run this 3 times so there are other INSERT events in the queue? Then
Done
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 20:17:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14362/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 22:50:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 02:47:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14174/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 22:18:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6:
(7 comments)
Thanks for the fixing the undefined behavior with hms event sync requests!
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1428
PS6, Line 1428: lastSyncedEventId_.set(getGreatestSyncedEventId());
nit: It's worth mentioning here that lastSycnedEventId is the last dispatched event id in hierarchical event processing
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1429
PS6, Line 1429: lastSyncedEventTimeSecs_.set(getGreatestSyncedEventTime());
Is it possible that another TableExecutor thread updates the processedLog_ of EventExecutorService concurrently after L1428 then this event time is inconsistent with the above event id? Maybe getting the pair of <eventId, eventTime> at once is more robust.
eventExecutorService_.clear() also uses its processedLog_. If they could be inconsistent, we can let clear() returns the pair of greatest synced event id + time and set them to lastSyncedEventId_ and lastSyncedEventTimeSecs_
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
File fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java@187
PS6, Line 187: Preconditions.checkState(state_.isDropProcessed());
nit: let's avoid using Preconditions checks without error messages, e.g.
Preconditions.checkState(state_.isDropProcessed(),
"Pseudo DROP event should have been processed");
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/4/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@473
PS4, Line 473: if (!waiter_.await(0)) break;
> Thanks, good catch. I had assumed invalidate metadata was the only way to r
Done
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
File fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java:
http://gerrit.cloudera.org:8080/#/c/24975/6/fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java@373
PS6, Line 373: // Throwing EventProcessException updates the event processor status to ERROR or
: // NEEDS_INVALIDATE, which clears EventExecutorService and thereby removes all
: // DbProcessors and TableProcessors. Global invalidate metadata is triggered
: // without user intervention iff
: // invalidate_global_metadata_on_event_processing_failure flag is true.
: // Otherwise, user has to issue invalidate metadata or start event processor.
nit: this appears 3 times. How about moving it to the class comment of EventProcessException?
http://gerrit.cloudera.org:8080/#/c/24975/6/tests/custom_cluster/test_event_processing_error.py
File tests/custom_cluster/test_event_processing_error.py:
http://gerrit.cloudera.org:8080/#/c/24975/6/tests/custom_cluster/test_event_processing_error.py@61
PS6, Line 61: self.run_stmt_in_hive("insert into {}.t1 values (1),(2),(3)".format(unique_database))
Can we run this 3 times so there are other INSERT events in the queue? Then after L78 they should all be discarded.
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py
File tests/custom_cluster/test_event_processing_perf.py:
http://gerrit.cloudera.org:8080/#/c/24975/4/tests/custom_cluster/test_event_processing_perf.py@82
PS4, Line 82: "--enable_hierarchical_event_processing=false")
> Done. All these tests exists with explicit "--enable_hierarchical_event_pro
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 23 Oct 2026 06:56:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 22:19:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 13:34:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14487/ DRY_RUN=true
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 19:16:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#5).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
1) When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
2) When an event fails to process, the event processor changes to ERROR
or NEEDS_INVALIDATE status and the executors stop processing, leaving
the failed event and the events after it queued on the executors.
These stale events were not discarded. So upon event processor start,
they were processed first, regardless of the event id the event
processor is started from.
To fix it, the events queued on the executors are now discarded when
the event processor changes to ERROR or NEEDS_INVALIDATE. The greatest
synced event is retained, and the last dispatched event is moved back
to it. Upon start, event processing resumes from the greatest synced
event id, the latest event id or the given event id, and the events
after it are fetched from HMS again.
3) HMS event sync requests in hierarchical mode now fail if the event
processor cannot process events, or the last dispatched event moves
back below the event id they wait for. Discarded events make their
databases and tables appear processed, so the requests could
otherwise succeed with stale metadata.
4) This also fixes a race in RenameTableBarrierEvent, where the pseudo
drop and create table event threads could both see both events
processed and skipped, and both increment the events-skipped metric.
Event-driven wakeups made it more likely, causing
test_rename_table_hierarchical_one_db_and_two_table_threads to fail
intermittently. Since the pseudo create table event is always
processed after the pseudo drop table event, only the create now
updates the skipped metric and the delete event log, and injects the
error for testing.
Testing:
- Ran existing tests.
- Added test_event_processor_start_upon_error to verify that the
failed event is retried upon :event_processor('start'), and is not
retried upon :event_processor('start', -1) even when a subsequent
event is enqueued for the same table.
- The non-hierarchical variants of the tests are now set
--enable_hierarchical_event_processing=false explicitly. The flag
defaults to true, so they were running in hierarchical mode.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_error.py
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 501 insertions(+), 150 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/5
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24599/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 07:00:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14301/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 08:17:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14363/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 31 Oct 2026 04:09:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#8).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
1) When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
2) When an event fails to process, the event processor changes to ERROR
or NEEDS_INVALIDATE status and the executors stop processing, leaving
the failed event and the events after it queued on the executors.
These stale events were not discarded. So upon event processor start,
they were processed first, regardless of the event id the event
processor is started from.
To fix it, the events queued on the executors are now discarded when
the event processor changes to ERROR or NEEDS_INVALIDATE. The greatest
synced event is retained, and the last dispatched event is moved back
to it. Upon start, event processing resumes from the greatest synced
event id, the latest event id or the given event id, and the events
after it are fetched from HMS again.
3) HMS event sync requests in hierarchical mode now fail if the event
processor cannot process events, or the event executors are cleared
while they wait, i.e. upon event processing failure or global
INVALIDATE METADATA. Clearing discards the queued events and makes
their databases and tables appear processed, so the requests could
otherwise succeed with stale metadata. EventExecutorService counts the
number of times it is cleared. A request captures the count before
checking its databases and tables, and fails if the count changes.
4) This also fixes a race in RenameTableBarrierEvent, where the pseudo
drop and create table event threads could both see both events
processed and skipped, and both increment the events-skipped metric.
Event-driven wakeups made it more likely, causing
test_rename_table_hierarchical_one_db_and_two_table_threads to fail
intermittently. Since the pseudo create table event is always
processed after the pseudo drop table event, only the create now
updates the skipped metric and the delete event log, and injects the
error for testing.
Testing:
- Ran existing tests.
- Added test_event_processor_start_upon_error to verify that the
failed event is retried upon :event_processor('start'), and is not
retried upon :event_processor('start', -1) even when a subsequent
event is enqueued for the same table. Multiple INSERT events are
fetched in a batch, so that the events queued after the failed
event are discarded upon failure.
- Added test_hms_event_sync_fails_upon_catalog_reset to verify that a
query waiting for HMS events to be synced fails when a global
INVALIDATE METADATA discards the events queued on the event
executors.
- The non-hierarchical variants of the tests are now set
--enable_hierarchical_event_processing=false explicitly. The flag
defaults to true, so they were running in hierarchical mode.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/EventProcessException.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
M tests/custom_cluster/test_event_processing_error.py
M tests/custom_cluster/test_event_processing_perf.py
M tests/custom_cluster/test_events_custom_configs.py
12 files changed, 588 insertions(+), 156 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/8
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 7:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24975/7/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/24975/7/tests/custom_cluster/test_events_custom_configs.py@2133
PS7, Line 2133: d
flake8: E306 expected 1 blank line before a nested definition, found 0
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 16:51:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9:
(3 comments)
Reworked for comments on PS8
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1429
PS8, Line 1429: clear(), since the pro
> nit: maybe more clear saying "and no events will be dispatched while this l
Done
https://gerrit.cloudera.org/#/c/24975/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@2181
PS8, Line 2181: }
> Is it possible that getGreatestSyncedEventId() has reached latestEventId at
Done. So instead of failing when the clear count changes, the request now falls back to waiting for the greatest synced event id to reach its latest event id. That's the same wait as in non-hierarchical mode, so both now share waitForEventSyncUpTo(). getSyncInterruptedError() is replaced by isSyncInterrupted(). The request still fails if the event processor goes to ERROR or NEEDS_INVALIDATE, or on timeout.
https://gerrit.cloudera.org/#/c/24975/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
https://gerrit.cloudera.org/#/c/24975/8/tests/custom_cluster/test_events_custom_configs.py@2122
PS8, Line 2122: client = self.default_impala_client(vector.get_value('protocol'))
> Shouldn't we let the query succeed since global INVALIDATE METADATA has cle
Done. Agreed. The reset invalidates all tables, and start(currentEventId) advances the greatest synced event id to at least the request's latest event id, so the query can succeed with the latest metadata, same as when HEP is disabled. With the fallback in the previous comment, the request now waits for that instead of failing.
Renamed the test to test_hms_event_sync_upon_catalog_reset and moved it to TestEventSyncWaiting, since it now expects success. It verifies the query gets the new row, and that catalogd logs the interruption to confirm the fallback path is taken. Also increased the event processing delay to 6s, so that the INSERT event is still pending when the reset clears the executors.
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 15 Nov 2026 02:11:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24249/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 22:40:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24257/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 09:05:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14198/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 11:40:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 11:46:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14301/
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 12:51:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 20:18:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14484/
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 18:24:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/24975 )
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
Patch Set 9: Verified+1
--
To view, visit https://gerrit.cloudera.org/24975
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 23:55:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15451: Make hierarchical event executors event-driven
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#2).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
Testing:
- Ran existing tests.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
8 files changed, 269 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/2
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>