You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2023/05/23 00:50:07 UTC

[Impala-ASF-CR] WIP IMPALA-12155: Support CatalogD HA

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19914


Change subject: WIP IMPALA-12155: Support CatalogD HA
......................................................................

WIP IMPALA-12155: Support CatalogD HA

To support catalog HA, we allow two catalog instances in an Active-
Passive HA pair to be added to an Impala cluster.
We add the preemptive behavior for catalogd. When enabled, the
preemptive behavior allows the catalogd with the higher priority to
become active and the paired catalogd becomes standby. The active
catalogd acts as the source of metadata and provides catalog service
for the Impala cluster.

To enable catalog HA for a cluster, two catalogds in the HA pair and
statestore must bestarted with starting flag "enable_catalog_ha".

The catalogd in an Active-Passive HA pair can be assigned an instance
priority value to indicate a preference for which catalogd should assume
the active role. The registration ID which is assigned by statestore can
be used as instance priority value. The lower numerical value in
registration ID corresponds to a higher priority. The catalogd with the
higher priority is designated as active, the other catalogd is
designated as standby. Only the active catalogd propagates the
IMPALA_CATALOG_TOPIC to the cluster. This guarantees only one writer for
the IMPALA_CATALOG_TOPIC in a Impala cluster.

statestore which is the registration center of an Impala cluster assigns
the roles for the catalogd in the HA pair after both catalogd register
to statestore. When statestore detects the active catalogd is not
healthy, it fails over catalog service to standby catalogd. When
failover occurs, statestore sends notifications with the address of
active catalogd to all coordinators and catalogd in the cluster. The
events are logged in the statestore and catalogd logs. When the catalogd
with the higher priority recovers from a failure, statestore does not
resume it as active to avoid flip-flop between the two catalogd.

To make a specific catalogd in the HA pair as active instance, the
catalogd must be started with starting flag "force_catalog_active" so
that the catalogd will be assigned with active role when it registers
to statestore. This allows administrator to manually perform catalog
service failover.

Testing:
 - Passed the core tests.
 - TODO unit-test

Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/exec/catalog-op-executor.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admissiond-env.cc
M be/src/service/client-request-state.cc
A be/src/statestore/statestore-subscriber-catalog.h
M be/src/statestore/statestore-subscriber-client-wrapper.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
16 files changed, 840 insertions(+), 84 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/19914/1
-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-12155: Support CatalogD HA

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19914 )

Change subject: IMPALA-12155: Support CatalogD HA
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13257/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Sun, 11 Jun 2023 23:19:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12155: Support CatalogD HA

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19914 )

Change subject: IMPALA-12155: Support CatalogD HA
......................................................................

IMPALA-12155: Support CatalogD HA

To support catalog HA, we allow two catalogd instances in an Active-
Passive HA pair to be added to an Impala cluster.
We add the preemptive behavior for catalogd. When enabled, the
preemptive behavior allows the catalogd with the higher priority to
become active and the paired catalogd becomes standby. The active
catalogd acts as the source of metadata and provides catalog service
for the Impala cluster.

To enable catalog HA for a cluster, two catalogds in the HA pair and
statestore must be started with starting flag "enable_catalogd_ha".

The catalogd in an Active-Passive HA pair can be assigned an instance
priority value to indicate a preference for which catalogd should assume
the active role. The registration ID which is assigned by statestore can
be used as instance priority value. The lower numerical value in
registration ID corresponds to a higher priority. The catalogd with the
higher priority is designated as active, the other catalogd is
designated as standby. Only the active catalogd propagates the
IMPALA_CATALOG_TOPIC to the cluster. This guarantees only one writer for
the IMPALA_CATALOG_TOPIC in a Impala cluster.

statestore which is the registration center of an Impala cluster assigns
the roles for the catalogd in the HA pair after both catalogd register
to statestore. When statestore detects the active catalogd is not
healthy, it fails over catalog service to standby catalogd. When
failover occurs, statestore sends notifications with the address of
active catalogd to all coordinators and catalogd in the cluster. The
events are logged in the statestore and catalogd logs. When the catalogd
with the higher priority recovers from a failure, statestore does not
resume it as active to avoid flip-flop between the two catalogd.

To make a specific catalogd in the HA pair as active instance, the
catalogd must be started with starting flag "force_catalogd_active" so
that the catalogd will be assigned with active role when it registers
to statestore. This allows administrator to manually perform catalog
service failover.

Added option "--enable_catalogd_ha" in bin/start-impala-cluster.py.
If the option is specified when running the script, the script will
create an Impala cluster with two catalogd instances in HA pair.

Testing:
 - Passed the core tests.
 - Added unit-test for auto failover.
 - TODO add more unit-tests, including manual failover.

Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalogd-main.cc
M be/src/common/global-flags.cc
M be/src/runtime/exec-env.cc
M be/src/statestore/statestore-subscriber-catalog.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M bin/start-impala-cluster.py
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
A tests/custom_cluster/test_catalogd_ha.py
17 files changed, 750 insertions(+), 132 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/19914/9
-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>

[Impala-ASF-CR] IMPALA-12155: Support CatalogD HA

Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.

Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/19914 )

Change subject: IMPALA-12155: Support CatalogD HA
......................................................................


Patch Set 8:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:

http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/catalog/catalog-server.h@151
PS8, Line 151:   /// Metric for Catalog Service port of this catalogd.
Please describe metric better.


http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/catalog/catalog-server.h@154
PS8, Line 154:   /// Metric that track if this catalogd is active when catalogd HA is enabled.
nit:tracks


http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/catalog/catalog-server.h@157
PS8, Line 157:   /// Metric to count the number of active status change.
nit:changes


http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/statestore/statestore.h
File be/src/statestore/statestore.h:

http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/statestore/statestore.h@768
PS8, Line 768:   /// Metric that track the address of active catalogd for catalogd HA
nit:tracks


http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/statestore/statestore.cc
File be/src/statestore/statestore.cc:

http://gerrit.cloudera.org:8080/#/c/19914/8/be/src/statestore/statestore.cc@557
PS8, Line 557:   active_catalogd_port_metric_ = metrics->AddGauge(STATESTORE_ACTIVE_CATALOGD_PORT, 0);
Why is port number tracked as a gauge rather than a property?



-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Tue, 13 Jun 2023 15:44:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12155: Support CatalogD HA

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19914 )

Change subject: IMPALA-12155: Support CatalogD HA
......................................................................

IMPALA-12155: Support CatalogD HA

To support catalog HA, we allow two catalogd instances in an Active-
Passive HA pair to be added to an Impala cluster.
We add the preemptive behavior for catalogd. When enabled, the
preemptive behavior allows the catalogd with the higher priority to
become active and the paired catalogd becomes standby. The active
catalogd acts as the source of metadata and provides catalog service
for the Impala cluster.

To enable catalog HA for a cluster, two catalogds in the HA pair and
statestore must be started with starting flag "enable_catalogd_ha".

The catalogd in an Active-Passive HA pair can be assigned an instance
priority value to indicate a preference for which catalogd should assume
the active role. The registration ID which is assigned by statestore can
be used as instance priority value. The lower numerical value in
registration ID corresponds to a higher priority. The catalogd with the
higher priority is designated as active, the other catalogd is
designated as standby. Only the active catalogd propagates the
IMPALA_CATALOG_TOPIC to the cluster. This guarantees only one writer for
the IMPALA_CATALOG_TOPIC in a Impala cluster.

statestore which is the registration center of an Impala cluster assigns
the roles for the catalogd in the HA pair after both catalogd register
to statestore. When statestore detects the active catalogd is not
healthy, it fails over catalog service to standby catalogd. When
failover occurs, statestore sends notifications with the address of
active catalogd to all coordinators and catalogd in the cluster. The
events are logged in the statestore and catalogd logs. When the catalogd
with the higher priority recovers from a failure, statestore does not
resume it as active to avoid flip-flop between the two catalogd.

To make a specific catalogd in the HA pair as active instance, the
catalogd must be started with starting flag "force_catalogd_active" so
that the catalogd will be assigned with active role when it registers
to statestore. This allows administrator to manually perform catalog
service failover.

Added option "--enable_catalogd_ha" in bin/start-impala-cluster.py.
If the option is specified when running the script, the script will
create an Impala cluster with two catalogd instances in HA pair.

Testing:
 - Passed the core tests.
 - Added unit-test for auto failover and manual failover.

Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalogd-main.cc
M be/src/common/global-flags.cc
M be/src/runtime/exec-env.cc
M be/src/statestore/statestore-subscriber-catalog.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M bin/start-impala-cluster.py
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
A tests/custom_cluster/test_catalogd_ha.py
17 files changed, 835 insertions(+), 133 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/19914/11
-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>

[Impala-ASF-CR] IMPALA-12155: Support High Availability for CatalogD

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19914 )

Change subject: IMPALA-12155: Support High Availability for CatalogD
......................................................................


Patch Set 17: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I68ce7e57014e2a01133aede7853a212d90688ddd
Gerrit-Change-Number: 19914
Gerrit-PatchSet: 17
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 11:36:31 +0000
Gerrit-HasComments: No