You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Petr Fedchenkov (Code Review)" <ge...@cloudera.org> on 2026/10/20 12:30:27 UTC
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Petr Fedchenkov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25153
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
IMPALA-15575: Choose the active statestored from the current stub state
StatestoreSubscriber::Start() sets the active statestored from the
answers of its two registrations. Both answers can say active and be
stale: if the active statestored fails over after the first registration
and before the second one, no UpdateStatestoredRole RPC reaches the
subscriber in between, and Start() follows the first statestored
although it is gone or standby. Similarly, after a failover the stub of
the standby statestored can keep a stale active flag, so the
subscriber's metrics show both statestoreds as active (IMPALA-15575).
This patch adds ReconcileRoles(), called at the end of Start() and after
every re-registration. It follows the registered stub that reports
active with the highest active version and clears a stale active flag on
the other stub. If no registered stub reports active, it keeps the
current choice, so that a role update from the other statestored is
still handled as a failover. When the followed statestored is elected
again, UpdateStatestoredRole() now also clears the standby flag of its
last registration. The DCHECK in Start() that at most one answer says
active is removed, since both can. The checks are turned off with
--statestore_subscriber_ha_role_checks=false.
Testing:
- Added test_statestored_failover_between_startup_registrations. It
failed with --statestore_subscriber_ha_role_checks=false (the
catalogd kept following the killed statestored) and passed 4 times
with the fix.
- Ran test_statestored_ha.py and test_catalogd_ha.py with the core
exploration strategy, and all tests passed.
Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Assisted-by: Claude Opus 5.5 (Claude Code)
---
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M tests/custom_cluster/test_statestored_ha.py
3 files changed, 140 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/25153/1
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 1
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25153 )
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24645/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 3
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 17:16:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Posted by "Petr Fedchenkov (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25153
to look at the new patch set (#3).
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
IMPALA-15575: Choose the active statestored from the current stub state
StatestoreSubscriber::Start() sets the active statestored from the
answers of its two registrations. Both answers can say active and be
stale: if the active statestored fails over after the first registration
and before the second one, no UpdateStatestoredRole RPC reaches the
subscriber in between, and Start() follows the first statestored
although it is gone or standby. Similarly, after a failover the stub of
the standby statestored can keep a stale active flag, so the
subscriber's metrics show both statestoreds as active (IMPALA-15575).
This patch adds ReconcileRoles(), called at the end of Start() and after
every re-registration. It follows the registered stub that reports
active with the highest active version and clears a stale active flag on
the other stub. If no registered stub reports active, it keeps the
current choice, so that a role update from the other statestored is
still handled as a failover. When the followed statestored is elected
again, UpdateStatestoredRole() now also clears the standby flag of its
last registration. The DCHECK in Start() that at most one answer says
active is removed, since both can. The failover swap in
UpdateStatestoredRole() now checks the roles again under the lock, so
that the recovery thread and the role RPC of the same statestored do not
both swap and switch back to the old statestored. The checks are turned
off with --statestore_subscriber_ha_role_checks=false.
Testing:
- Added test_statestored_failover_between_startup_registrations. It
failed with --statestore_subscriber_ha_role_checks=false (the
catalogd kept following the killed statestored) and passed 4 times
with the fix.
- Ran test_statestored_ha.py and test_catalogd_ha.py with the core
exploration strategy, and all tests passed.
Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Assisted-by: Claude Opus 5.5 (Claude Code)
---
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M tests/custom_cluster/test_statestored_ha.py
3 files changed, 154 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/25153/3
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 3
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Posted by "Petr Fedchenkov (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25153
to look at the new patch set (#2).
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
IMPALA-15575: Choose the active statestored from the current stub state
StatestoreSubscriber::Start() sets the active statestored from the
answers of its two registrations. Both answers can say active and be
stale: if the active statestored fails over after the first registration
and before the second one, no UpdateStatestoredRole RPC reaches the
subscriber in between, and Start() follows the first statestored
although it is gone or standby. Similarly, after a failover the stub of
the standby statestored can keep a stale active flag, so the
subscriber's metrics show both statestoreds as active (IMPALA-15575).
This patch adds ReconcileRoles(), called at the end of Start() and after
every re-registration. It follows the registered stub that reports
active with the highest active version and clears a stale active flag on
the other stub. If no registered stub reports active, it keeps the
current choice, so that a role update from the other statestored is
still handled as a failover. When the followed statestored is elected
again, UpdateStatestoredRole() now also clears the standby flag of its
last registration. The DCHECK in Start() that at most one answer says
active is removed, since both can. The checks are turned off with
--statestore_subscriber_ha_role_checks=false.
Testing:
- Added test_statestored_failover_between_startup_registrations. It
failed with --statestore_subscriber_ha_role_checks=false (the
catalogd kept following the killed statestored) and passed 4 times
with the fix.
- Ran test_statestored_ha.py and test_catalogd_ha.py with the core
exploration strategy, and all tests passed.
Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Assisted-by: Claude Opus 5.5 (Claude Code)
---
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M tests/custom_cluster/test_statestored_ha.py
3 files changed, 140 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/25153/2
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 2
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25153 )
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24603/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 1
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 12:55:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15575: Choose the active statestored from the current stub state
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25153 )
Change subject: IMPALA-15575: Choose the active statestored from the current stub state
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24608/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f0c999a0eb4c8049b97223aba902dd55da708f0
Gerrit-Change-Number: 25153
Gerrit-PatchSet: 2
Gerrit-Owner: Petr Fedchenkov <gi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 13:26:14 +0000
Gerrit-HasComments: No