You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by santi kumar <sa...@gmail.com> on 2012/12/11 11:32:36 UTC
Cassandra on EC2 - describe_ring() is giving private IPs
We have a 4 node cluster in us-east region in two different AZ's. Clients
connect to this cluster from our datacenter which is not on AWS.
Hector clients are initialized with public DNS names, then listern_address
is with private ip and rpc_address is with 0.0.0.0.
Having issues with Node Auto Discovery by Hector. When it's trying to
discover the ring, the end points are initialized with private IP's for all
Token Ranges. It checks with the existing hosts (which are initialized
public DNS) and thinks that there is a new node got added to the cluster.
looks like describe_ring() is the one which provides the details. Though I
configured the listen_address with public dns, still I had the same issue.
Any idea, what is the best way to configure for EC2. Have gone through the
link
https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
But not sure whether it's fixed in 1.1.4. When I run the nodetool ring, it
gives the private ips. But in the above doc, it shows the public IPs as
part of nodetool ring.
Some insight into this is really helpful.
Thanks
Santi
Re: Cassandra on EC2 - describe_ring() is giving private IPs
Posted by santi kumar <sa...@gmail.com>.
Yes That worked. Thanks for the pointer. Once the broadcast_address is
pointed to public IP, end points are coming with public IP. so Hectors
NodeAutoDiscoveryService matches with the existing host and not treating it
as new node.
On Wed, Dec 12, 2012 at 11:10 PM, Andrey Ilinykh <ai...@gmail.com> wrote:
> It makes sense. rpc_address is interface to listen. Try to set up public
> IP to broadcast_address.
>
> Andrey
>
>
> On Wed, Dec 12, 2012 at 9:33 AM, santi kumar <sa...@gmail.com>wrote:
>
>> When I configured rpc_address with public IP, cassandra is not starting
>> up. It's trowing 'unable to create thrift socket on <public IP>. When I
>> changed it to private IP, it was good.
>>
>> java.lang.RuntimeException: Unable to create thrift socket to /
>> 107.21.80.94:9160
>> at
>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:148)
>> at
>> org.apache.cassandra.thrift.CassandraDaemon.startServer(CassandraDaemon.java:76)
>> at
>> org.apache.cassandra.service.AbstractCassandraDaemon.startRPCServer(AbstractCassandraDaemon.java:300)
>> at
>> org.apache.cassandra.service.AbstractCassandraDaemon.start(AbstractCassandraDaemon.java:272)
>> at
>> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:369)
>> at
>> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
>> Caused by: org.apache.thrift.transport.TTransportException: Could not
>> create ServerSocket on address /107.21.80.94:9160.
>> at
>> org.apache.cassandra.thrift.TCustomServerSocket.<init>(TCustomServerSocket.java:80)
>> at
>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:141)
>>
>>
>>
>> On Wed, Dec 12, 2012 at 3:34 AM, aaron morton <aa...@thelastpickle.com>wrote:
>>
>>> Though I configured the listen_address with public dns, still I had the
>>> same issue.
>>>
>>> Internally the public DNS resolves to the private IP.
>>>
>>> looks like describe_ring() is the one which provides the details.
>>>
>>> describe_ring() returns includes the registered RPC addresses for the
>>> nodes. Trying setting the rpc_address to the public IP.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 11/12/2012, at 11:32 PM, santi kumar <sa...@gmail.com> wrote:
>>>
>>> We have a 4 node cluster in us-east region in two different AZ's.
>>> Clients connect to this cluster from our datacenter which is not on AWS.
>>>
>>> Hector clients are initialized with public DNS names, then
>>> listern_address is with private ip and rpc_address is with 0.0.0.0.
>>>
>>> Having issues with Node Auto Discovery by Hector. When it's trying to
>>> discover the ring, the end points are initialized with private IP's for all
>>> Token Ranges. It checks with the existing hosts (which are initialized
>>> public DNS) and thinks that there is a new node got added to the cluster.
>>>
>>> looks like describe_ring() is the one which provides the details. Though
>>> I configured the listen_address with public dns, still I had the same
>>> issue.
>>>
>>> Any idea, what is the best way to configure for EC2. Have gone through
>>> the link
>>>
>>> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>>>
>>> But not sure whether it's fixed in 1.1.4. When I run the nodetool ring,
>>> it gives the private ips. But in the above doc, it shows the public IPs as
>>> part of nodetool ring.
>>>
>>> Some insight into this is really helpful.
>>>
>>> Thanks
>>> Santi
>>>
>>>
>>>
>>
>
Re: Cassandra on EC2 - describe_ring() is giving private IPs
Posted by Andrey Ilinykh <ai...@gmail.com>.
It makes sense. rpc_address is interface to listen. Try to set up public IP
to broadcast_address.
Andrey
On Wed, Dec 12, 2012 at 9:33 AM, santi kumar <sa...@gmail.com> wrote:
> When I configured rpc_address with public IP, cassandra is not starting
> up. It's trowing 'unable to create thrift socket on <public IP>. When I
> changed it to private IP, it was good.
>
> java.lang.RuntimeException: Unable to create thrift socket to /
> 107.21.80.94:9160
> at
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:148)
> at
> org.apache.cassandra.thrift.CassandraDaemon.startServer(CassandraDaemon.java:76)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.startRPCServer(AbstractCassandraDaemon.java:300)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.start(AbstractCassandraDaemon.java:272)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:369)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
> Caused by: org.apache.thrift.transport.TTransportException: Could not
> create ServerSocket on address /107.21.80.94:9160.
> at
> org.apache.cassandra.thrift.TCustomServerSocket.<init>(TCustomServerSocket.java:80)
> at
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:141)
>
>
>
> On Wed, Dec 12, 2012 at 3:34 AM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> Though I configured the listen_address with public dns, still I had the
>> same issue.
>>
>> Internally the public DNS resolves to the private IP.
>>
>> looks like describe_ring() is the one which provides the details.
>>
>> describe_ring() returns includes the registered RPC addresses for the
>> nodes. Trying setting the rpc_address to the public IP.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 11/12/2012, at 11:32 PM, santi kumar <sa...@gmail.com> wrote:
>>
>> We have a 4 node cluster in us-east region in two different AZ's. Clients
>> connect to this cluster from our datacenter which is not on AWS.
>>
>> Hector clients are initialized with public DNS names, then
>> listern_address is with private ip and rpc_address is with 0.0.0.0.
>>
>> Having issues with Node Auto Discovery by Hector. When it's trying to
>> discover the ring, the end points are initialized with private IP's for all
>> Token Ranges. It checks with the existing hosts (which are initialized
>> public DNS) and thinks that there is a new node got added to the cluster.
>>
>> looks like describe_ring() is the one which provides the details. Though
>> I configured the listen_address with public dns, still I had the same
>> issue.
>>
>> Any idea, what is the best way to configure for EC2. Have gone through
>> the link
>>
>> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>>
>> But not sure whether it's fixed in 1.1.4. When I run the nodetool ring,
>> it gives the private ips. But in the above doc, it shows the public IPs as
>> part of nodetool ring.
>>
>> Some insight into this is really helpful.
>>
>> Thanks
>> Santi
>>
>>
>>
>
Re: Cassandra on EC2 - describe_ring() is giving private IPs
Posted by santi kumar <sa...@gmail.com>.
When I configured rpc_address with public IP, cassandra is not starting up.
It's trowing 'unable to create thrift socket on <public IP>. When I changed
it to private IP, it was good.
java.lang.RuntimeException: Unable to create thrift socket to /
107.21.80.94:9160
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:148)
at
org.apache.cassandra.thrift.CassandraDaemon.startServer(CassandraDaemon.java:76)
at
org.apache.cassandra.service.AbstractCassandraDaemon.startRPCServer(AbstractCassandraDaemon.java:300)
at
org.apache.cassandra.service.AbstractCassandraDaemon.start(AbstractCassandraDaemon.java:272)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:369)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
Caused by: org.apache.thrift.transport.TTransportException: Could not
create ServerSocket on address /107.21.80.94:9160.
at
org.apache.cassandra.thrift.TCustomServerSocket.<init>(TCustomServerSocket.java:80)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.<init>(CassandraDaemon.java:141)
On Wed, Dec 12, 2012 at 3:34 AM, aaron morton <aa...@thelastpickle.com>wrote:
> Though I configured the listen_address with public dns, still I had the
> same issue.
>
> Internally the public DNS resolves to the private IP.
>
> looks like describe_ring() is the one which provides the details.
>
> describe_ring() returns includes the registered RPC addresses for the
> nodes. Trying setting the rpc_address to the public IP.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 11/12/2012, at 11:32 PM, santi kumar <sa...@gmail.com> wrote:
>
> We have a 4 node cluster in us-east region in two different AZ's. Clients
> connect to this cluster from our datacenter which is not on AWS.
>
> Hector clients are initialized with public DNS names, then listern_address
> is with private ip and rpc_address is with 0.0.0.0.
>
> Having issues with Node Auto Discovery by Hector. When it's trying to
> discover the ring, the end points are initialized with private IP's for all
> Token Ranges. It checks with the existing hosts (which are initialized
> public DNS) and thinks that there is a new node got added to the cluster.
>
> looks like describe_ring() is the one which provides the details. Though I
> configured the listen_address with public dns, still I had the same issue.
>
> Any idea, what is the best way to configure for EC2. Have gone through the
> link
>
> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>
> But not sure whether it's fixed in 1.1.4. When I run the nodetool ring, it
> gives the private ips. But in the above doc, it shows the public IPs as
> part of nodetool ring.
>
> Some insight into this is really helpful.
>
> Thanks
> Santi
>
>
>
Re: Cassandra on EC2 - describe_ring() is giving private IPs
Posted by aaron morton <aa...@thelastpickle.com>.
> Though I configured the listen_address with public dns, still I had the same issue.
Internally the public DNS resolves to the private IP.
> looks like describe_ring() is the one which provides the details.
describe_ring() returns includes the registered RPC addresses for the nodes. Trying setting the rpc_address to the public IP.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 11/12/2012, at 11:32 PM, santi kumar <sa...@gmail.com> wrote:
> We have a 4 node cluster in us-east region in two different AZ's. Clients connect to this cluster from our datacenter which is not on AWS.
>
> Hector clients are initialized with public DNS names, then listern_address is with private ip and rpc_address is with 0.0.0.0.
>
> Having issues with Node Auto Discovery by Hector. When it's trying to discover the ring, the end points are initialized with private IP's for all Token Ranges. It checks with the existing hosts (which are initialized public DNS) and thinks that there is a new node got added to the cluster.
>
> looks like describe_ring() is the one which provides the details. Though I configured the listen_address with public dns, still I had the same issue.
>
> Any idea, what is the best way to configure for EC2. Have gone through the link
> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>
> But not sure whether it's fixed in 1.1.4. When I run the nodetool ring, it gives the private ips. But in the above doc, it shows the public IPs as part of nodetool ring.
>
> Some insight into this is really helpful.
>
> Thanks
> Santi