[jira] [Commented] (IMPALA-12060) statestore should only allow a single catalog instance to be part of cluster

["Wenzhe Zhou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718367#comment-17718367 ] 

Wenzhe Zhou commented on IMPALA-12060:
--------------------------------------

Submitted a patch: https://gerrit.cloudera.org/#/c/19826/
Currently the DNS resolution are done on subscriber's side, not on statestore.  Subscribers (coordinator/executor/catalogd) do DNS resolution if "FLAGS_statestore_subscriber_use_resolved_address" is set as true, then send the resolved heart-beat address to statestore.  statestore don't have host-name of the subscribers and cannot do DN resolution now.
In the patch, the DNS resolution are still done on subscriber's side. But each subscriber need to re-do DNS resolution when it try to re-register to statestore in recovery mode. If DNS resolution fails, it will not re-register itself to statestore. This prevents the catalogd in passive mode (original leader) from rejoining the cluster. When catalogd leader failover happens, new leader catalogd could do DNS resolution, and send registration to statestore. Then statestore un-register original leader catalogd before register new leader catalogd to ensure only one catalogd instance in the cluster.

Another approach is to move DNS resolution from subscriber to statestore so that statestore have host-name of the subscribers and could do DNS resolution. statestore will try to do DNS resolution with hostname of the subscribers when it try to register a subscriber. If DNS resolution fails, statestore will reject the registration request.

> statestore should only allow a single catalog instance to be part of cluster
> ----------------------------------------------------------------------------
>
>                 Key: IMPALA-12060
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12060
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Abhishek Rawat
>            Assignee: Wenzhe Zhou
>            Priority: Critical
>
> Today Impala doesn't support multiple catalogd instances to be part of the cluster. When using external Active Passive HA mechanisms like leader election, it may be possible for multiple catalog replicas to be part of the cluster, in some cases. This is mainly because catalogds have no idea that there are other catalogd instances. One way to address this issue could be for statestore to always ensure that there could only be a single instance of catalogd in the cluster. Coordinator also has a config `catalog_service_host` - both statestore and coordinator should have a consistent view and consider the proper catalogd instance to be part of the cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org