[jira] [Commented] (KUDU-1620) Consensus peer proxy hostnames should be reresolved on failure

["Grant Henke (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875253#comment-16875253 ] 

Grant Henke commented on KUDU-1620:
-----------------------------------

There was a discussion in the slack channel discussing how this impacts Kubernetes/Docker deployments. 

If a container/instance/host running a master is down/crashed tservers are unable to restart and fail with something like:
{noformat}
Tservers start crashing with the logs saying 'Couldn't resolve master service address 'master-1': unable to resolve address for master-1: Temporary failure in name resolution'. 
{noformat}

[~tlipcon] mentioned an option for a workaround in the thread:
{quote}
ah, maybe we will stay up if the master resolution fails at runtime, but will refuse to start if one of the masters is unresolvable.
which is probably on purpose to avoid people starting with typos in their master list and not noticing
for the k8s use case I can see why you might want it, though -- perhaps we need some flag to explicitly allow starting when some number o the masters are unresolvable, and be sure it keeps retrying to resolve
{quote}

This also appears to be why we added a 2 second delay to the docker entrypoint script: https://github.com/apache/kudu/blob/ad798391fdf22c1632a641dbb6be80085636602a/docker/kudu-entrypoint.sh#L80



 

> Consensus peer proxy hostnames should be reresolved on failure
> --------------------------------------------------------------
>
>                 Key: KUDU-1620
>                 URL: https://issues.apache.org/jira/browse/KUDU-1620
>             Project: Kudu
>          Issue Type: Bug
>          Components: consensus
>    Affects Versions: 1.0.0
>            Reporter: Adar Dembo
>            Priority: Major
>              Labels: docker
>
> Noticed this while documenting the workflow to replace a dead master, which currently bypasses Raft config changes in favor of having the replacement master "masquerade" as the dead master via DNS changes.
> Internally we never rebuild consensus peer proxies in the event of network failure; we assume that the peer will return at the same location. Nominally this is reasonable; allowing peers to change host/port information on the fly is tricky and has yet to be implemented. But, we should at least retry the DNS resolution; not doing so forces the workflow to include steps to restart the existing masters, which creates a (small) availability outage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)