You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Amir Dafny-Man <A....@f5.com> on 2016/09/12 06:35:18 UTC
Multiple Network Interfaces in non-EC2
Hi,
I followed the docs (http://docs.datastax.com/en/cassandra/3.x/cassandra/configuration/configMultiNetworks.html) and I am familiar with https://issues.apache.org/jira/browse/CASSANDRA-9748.
Trying to establish a single 2 nodes, single DC setup. I tried Cassandra 3.7 but also 2.2.7.
My setup:
RHEL6 distribution
Node1 external: 10.240.33.241
Node1 internal: 192.168.33.241
Node2 external: 10.240.33.244
Node2 internal: 192.168.33.244
cassandra-rackdc.properties (for both nodes) also tried with prefer_local=false:
dc=vdra015-xs-15
rack=rack1
prefer_local=true
Cassandra.yaml (changes over default):
seeds: "10.240.33.241"
listen_address: 192.168.33.241 or 192.168.33.244
broadcast_address: 10.240.33.241 or 10.240.33.244
listen_on_broadcast_address: true
rpc_address: 192.168.33.241 or 192.168.33.244
endpoint_snitch: GossipingPropertyFileSnitch
Routing table:
# ip r
192.168.33.0/24 dev eth1 proto kernel scope link src 192.168.33.241
10.1.21.0/24 dev eth2 proto kernel scope link src 10.1.21.241
10.1.22.0/24 dev eth3 proto kernel scope link src 10.1.22.241
10.1.23.0/24 dev eth4 proto kernel scope link src 10.1.23.241
10.240.32.0/21 dev eth0 proto kernel scope link src 10.240.33.241
default via 10.240.32.1 dev eth0
Experienced behavior:
1. Node1 starts up normally
# netstat -anlp|grep java
tcp 0 0 127.0.0.1:55452 0.0.0.0:* LISTEN 10036/java
tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN 10036/java
tcp 0 0 10.240.33.241:7000 0.0.0.0:* LISTEN 10036/java
tcp 0 0 192.168.33.241:7000 0.0.0.0:* LISTEN 10036/java
tcp 0 0 ::ffff:192.168.33.241:9042 :::* LISTEN 10036/java
2. When I try to start node2, it is unable to connect to node1 IP set in seeds
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds
java.lang.RuntimeException: Unable to gossip with any seeds
3. Running tcpdumpon node2, I can see that node2 is trying to connect to node1 external IP but with its source internal IP
# tcpdump -nn -i eth0 port 7000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:29:05.239026 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65015480 ecr 0,nop,wscale 9], length 0
09:29:06.238188 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65016480 ecr 0,nop,wscale 9], length 0
09:29:08.238159 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65018480 ecr 0,nop,wscale 9], length 0
09:29:12.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65022480 ecr 0,nop,wscale 9], length 0
09:29:20.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65030480 ecr 0,nop,wscale 9], length 0
09:29:36.238161 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65046480 ecr 0,nop,wscale 9], length 0
4. Running tcpdump on node1, shows packets are not arriving
Is something configured wrong or is this a bug?
Thanks,
Amir
Re: Multiple Network Interfaces in non-EC2
Posted by Anuj Wadehra <an...@yahoo.co.in>.
Hi Amir,
I would like to understand your requirement first. Why do you need multiface interface configuration mentioned at http://docs.datastax.com/en/cassandra/3.x/cassandra/configuration/configMultiNetworks.html with single DC setup?
As per my understanding, you could simply set listen address to private IP and dont set broadcast_address and listen_on_broadcast address properties at all. You could use your private IP everywhere because you dont have any other DC which would connect using public IP.
In multiple DCs, you need public IP for communicating with nodes in other DCs and thats where you need private IP for internal communication and public IP for across DC communication.
Let me know if using private IP solves your problem.
Also, if you have a specific use case for using multiple interface configuration, you could add a NAT rule to route your traffic on public IP to your private IP (route traffic on Cassandra port only). This could act as a workaround till the JIRA is fixed. Let me know if you see any issues with the workaround.
Thanks
Anuj
RE: Multiple Network Interfaces in non-EC2
Posted by Amir Dafny-Man <A....@f5.com>.
Hi Paulo,
There you go https://issues.apache.org/jira/browse/CASSANDRA-12631
Tell me if there is more info I can provide/test fix.
I wonder how the original CASSANDRA-9748 get closed?
I also try the setup on 2.2.5 which is one of the fix versions without success. Did this setup ever work?
Thanks,
Amir
From: Paulo Motta [mailto:pauloricardomg@gmail.com]
Sent: Monday, September 12, 2016 16:23
To: user@cassandra.apache.org
Subject: Re: Multiple Network Interfaces in non-EC2
This seems like a bug, it seems we always bind the outgoing socket to the private/listen address. Would you mind opening a JIRA and posting the link here?
Thanks
2016-09-12 3:35 GMT-03:00 Amir Dafny-Man <A....@f5.com>>:
Hi,
I followed the docs (http://docs.datastax.com/en/cassandra/3.x/cassandra/configuration/configMultiNetworks.html) and I am familiar with https://issues.apache.org/jira/browse/CASSANDRA-9748.
Trying to establish a single 2 nodes, single DC setup. I tried Cassandra 3.7 but also 2.2.7.
My setup:
RHEL6 distribution
Node1 external: 10.240.33.241
Node1 internal: 192.168.33.241
Node2 external: 10.240.33.244
Node2 internal: 192.168.33.244
cassandra-rackdc.properties (for both nodes) also tried with prefer_local=false:
dc=vdra015-xs-15
rack=rack1
prefer_local=true
Cassandra.yaml (changes over default):
seeds: “10.240.33.241”
listen_address: 192.168.33.241 or 192.168.33.244
broadcast_address: 10.240.33.241 or 10.240.33.244
listen_on_broadcast_address: true
rpc_address: 192.168.33.241 or 192.168.33.244
endpoint_snitch: GossipingPropertyFileSnitch
Routing table:
# ip r
192.168.33.0/24<http://192.168.33.0/24> dev eth1 proto kernel scope link src 192.168.33.241
10.1.21.0/24<http://10.1.21.0/24> dev eth2 proto kernel scope link src 10.1.21.241
10.1.22.0/24<http://10.1.22.0/24> dev eth3 proto kernel scope link src 10.1.22.241
10.1.23.0/24<http://10.1.23.0/24> dev eth4 proto kernel scope link src 10.1.23.241
10.240.32.0/21<http://10.240.32.0/21> dev eth0 proto kernel scope link src 10.240.33.241
default via 10.240.32.1 dev eth0
Experienced behavior:
1. Node1 starts up normally
# netstat -anlp|grep java
tcp 0 0 127.0.0.1:55452<http://127.0.0.1:55452> 0.0.0.0:* LISTEN 10036/java
tcp 0 0 127.0.0.1:7199<http://127.0.0.1:7199> 0.0.0.0:* LISTEN 10036/java
tcp 0 0 10.240.33.241:7000<http://10.240.33.241:7000> 0.0.0.0:* LISTEN 10036/java
tcp 0 0 192.168.33.241:7000<http://192.168.33.241:7000> 0.0.0.0:* LISTEN 10036/java
tcp 0 0 ::ffff:192.168.33.241:9042<http://192.168.33.241:9042> :::* LISTEN 10036/java
2. When I try to start node2, it is unable to connect to node1 IP set in seeds
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds
java.lang.RuntimeException: Unable to gossip with any seeds
3. Running tcpdumpon node2, I can see that node2 is trying to connect to node1 external IP but with its source internal IP
# tcpdump -nn -i eth0 port 7000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:29:05.239026 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65015480 ecr 0,nop,wscale 9], length 0
09:29:06.238188 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65016480 ecr 0,nop,wscale 9], length 0
09:29:08.238159 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65018480 ecr 0,nop,wscale 9], length 0
09:29:12.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65022480 ecr 0,nop,wscale 9], length 0
09:29:20.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65030480 ecr 0,nop,wscale 9], length 0
09:29:36.238161 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S], seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65046480 ecr 0,nop,wscale 9], length 0
4. Running tcpdump on node1, shows packets are not arriving
Is something configured wrong or is this a bug?
Thanks,
Amir
Re: Multiple Network Interfaces in non-EC2
Posted by Paulo Motta <pa...@gmail.com>.
This seems like a bug, it seems we always bind the outgoing socket to the
private/listen address. Would you mind opening a JIRA and posting the link
here?
Thanks
2016-09-12 3:35 GMT-03:00 Amir Dafny-Man <A....@f5.com>:
> Hi,
>
>
>
> I followed the docs (http://docs.datastax.com/en/cassandra/3.x/cassandra/
> configuration/configMultiNetworks.html) and I am familiar with
> https://issues.apache.org/jira/browse/CASSANDRA-9748.
>
> Trying to establish a single 2 nodes, single DC setup. I tried Cassandra
> 3.7 but also 2.2.7.
>
> My setup:
>
> RHEL6 distribution
>
> Node1 external: 10.240.33.241
>
> Node1 internal: 192.168.33.241
>
> Node2 external: 10.240.33.244
>
> Node2 internal: 192.168.33.244
>
>
>
> cassandra-rackdc.properties (for both nodes) also tried with
> prefer_local=false:
>
> dc=vdra015-xs-15
>
> rack=rack1
>
> prefer_local=true
>
>
>
> Cassandra.yaml (changes over default):
>
> seeds: “10.240.33.241”
>
> listen_address: 192.168.33.241 or 192.168.33.244
>
> broadcast_address: 10.240.33.241 or 10.240.33.244
>
> listen_on_broadcast_address: true
>
> rpc_address: 192.168.33.241 or 192.168.33.244
>
> endpoint_snitch: GossipingPropertyFileSnitch
>
>
>
> Routing table:
>
> # ip r
>
> 192.168.33.0/24 dev eth1 proto kernel scope link src 192.168.33.241
>
> 10.1.21.0/24 dev eth2 proto kernel scope link src 10.1.21.241
>
> 10.1.22.0/24 dev eth3 proto kernel scope link src 10.1.22.241
>
> 10.1.23.0/24 dev eth4 proto kernel scope link src 10.1.23.241
>
> 10.240.32.0/21 dev eth0 proto kernel scope link src 10.240.33.241
>
> default via 10.240.32.1 dev eth0
>
>
>
> Experienced behavior:
>
> 1. Node1 starts up normally
>
> # netstat -anlp|grep java
>
> tcp 0 0 127.0.0.1:55452 0.0.0.0:*
> LISTEN 10036/java
>
> tcp 0 0 127.0.0.1:7199 0.0.0.0:*
> LISTEN 10036/java
>
> tcp 0 0 10.240.33.241:7000 0.0.0.0:*
> LISTEN 10036/java
>
> tcp 0 0 192.168.33.241:7000 0.0.0.0:*
> LISTEN 10036/java
>
> tcp 0 0 ::ffff:192.168.33.241:9042
> :::* LISTEN 10036/java
>
> 2. When I try to start node2, it is unable to connect to node1 IP
> set in seeds
>
> Exception (java.lang.RuntimeException) encountered during startup: Unable
> to gossip with any seeds
>
> java.lang.RuntimeException: Unable to gossip with any seeds
>
> 3. Running tcpdumpon node2, I can see that node2 is trying to
> connect to node1 external IP but with its source internal IP
>
> # tcpdump -nn -i eth0 port 7000
>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>
> 09:29:05.239026 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65015480 ecr
> 0,nop,wscale 9], length 0
>
> 09:29:06.238188 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65016480 ecr
> 0,nop,wscale 9], length 0
>
> 09:29:08.238159 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65018480 ecr
> 0,nop,wscale 9], length 0
>
> 09:29:12.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65022480 ecr
> 0,nop,wscale 9], length 0
>
> 09:29:20.238129 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65030480 ecr
> 0,nop,wscale 9], length 0
>
> 09:29:36.238161 IP 192.168.33.244.52900 > 10.240.33.241.7000: Flags [S],
> seq 77957108, win 14600, options [mss 1460,sackOK,TS val 65046480 ecr
> 0,nop,wscale 9], length 0
>
> 4. Running tcpdump on node1, shows packets are not arriving
>
>
>
> Is something configured wrong or is this a bug?
>
>
>
> Thanks,
>
> *Amir*
>
>
>