[jira] [Commented] (CASSANDRA-16718) Changing listen_address with prefer_local may lead to issues

["Simon Fontana Oscarsson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407896#comment-17407896 ] 

Simon Fontana Oscarsson commented on CASSANDRA-16718:
-----------------------------------------------------

Hi [~brandon.williams],

Jan asked me to answer since I have a deeper knowledge in the issue at hand.

We deploy Cassandra in a Kubernetes cluster. To achieve external connectivity to remote DCs (Cassandra DCs in different Kubernetes clusters) we use MetalLB (one LoadBalancer Service per Pod). Some of our deployments use traffic policy "Cluster" which doesn't preserve source IP.
{quote}However, kube-proxy will obscure the source IP address of the connection when it does load balancing, so your pod logs will show that external traffic appears to be coming from the service’s leader node.
{quote}
- [https://metallb.universe.tf/usage/#cluster-traffic-policy]

We have a Cassandra plugin which uses the IP for its execution logic. With prefer_local=false all traffic (incl. local traffic) will go through the load balancer and therefore obscures the source IP. With prefer_local the source IP for local nodes is preserved so it's available to our plugin.

Let me know if you need more details.

> Changing listen_address with prefer_local may lead to issues
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-16718
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16718
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Config
>            Reporter: Jan Karlsson
>            Priority: Normal
>             Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Many container based solution function by assigning new listen_addresses when nodes are stopped. Changing the listen_address is usually as simple as turning off the node and changing the yaml file. 
> However, if prefer_local is enabled, I observed that nodes were unable to join the cluster and fail with 'Unable to gossip with any seeds'. 
> Trace shows that the changing node will try to communicate with the existing node but the response is never received. I assume it is because the existing node attempts to communicate with the local address during the shadow round.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org