You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2023/04/21 18:52:00 UTC
[jira] [Comment Edited] (IMPALA-12060) statestore should only allow a single catalog instance to be part of cluster
[ https://issues.apache.org/jira/browse/IMPALA-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715129#comment-17715129 ]
Wenzhe Zhou edited comment on IMPALA-12060 at 4/21/23 6:51 PM:
---------------------------------------------------------------
Currently Statestore server does not distinguish the role of each subscriber, e.g. it does not distinguish if a subscriber is a catalogd, or impalad, or admissiond. Each daemon must register itself to statestored with an unique subscriber-id.
catalogd set its subscriber-id as "catalog-server@<hostname>:<port>".
impalad set its subscriber-id as "impalad@<hostname>:<port>".
admissiond set its subscriber-id as "admissiond@<hostname>:<port>".
If two daemons register to statestore with same subscriber-id, statestore removes the first one from the subscriber list and adds the second one with new registration_id. The first one will not receiver heartbeat and topic update RPCs.
If two daemons with different hostnames but same network addresses, statestore still accept the registration requests as two different subscribers.
To ensure there could only be a single instance of catalogd in the cluster, statestore have to distinguish the role of each subscriber so that it can count the number of instance for catalogd. We can find the role of the subscriber by checking the prefix of subscriber-id, or adding a new field in TRegisterSubscriberRequest.
was (Author: wzhou):
Currently Statestore server does not distinguish the role of each subscriber, e.g. it does not distinguish if a subscriber is a catalogd, or impalad, or admissiond. Each daemon must register itself to statestored with an unique subscriber-id.
catalogd set its subscriber-id as "catalog-server@<hostname>:<port>".
impalad set its subscriber-id as "impalad@<hostname>:<port>".
admissiond set its subscriber-id as "admissiond@<hostname>:<port>".
If two daemons register to statestore with same subscriber-id, statestore removes the first one from the subscriber list and adds the second one with new registration_id. The first one will not receiver heartbeat and topic update RPCs.
If two daemons with different hostnames but same network addresses, statestore still accept the registration requests as two different subscribers.
To ensure there could only be a single instance of catalogd in the cluster, statestore have to distinguish the role of each subscriber so that it counts the number of instance for catalogd. We can find the role of the subscriber by checking the prefix of subscriber-id, or adding a new field in TRegisterSubscriberRequest.
> statestore should only allow a single catalog instance to be part of cluster
> ----------------------------------------------------------------------------
>
> Key: IMPALA-12060
> URL: https://issues.apache.org/jira/browse/IMPALA-12060
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Assignee: Wenzhe Zhou
> Priority: Critical
>
> Today Impala doesn't support multiple catalogd instances to be part of the cluster. When using external Active Passive HA mechanisms like leader election, it may be possible for multiple catalog replicas to be part of the cluster, in some cases. This is mainly because catalogds have no idea that there are other catalogd instances. One way to address this issue could be for statestore to always ensure that there could only be a single instance of catalogd in the cluster. Coordinator also has a config `catalog_service_host` - both statestore and coordinator should have a consistent view and consider the proper catalogd instance to be part of the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org