You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jai Bheemsen Rao Dhanwada <ja...@gmail.com> on 2016/10/20 20:12:42 UTC
Rebuild failing while adding new datacenter
Hello All,
I have single datacenter with 3 C* nodes and we are trying to expand the
cluster to another region/DC. I am seeing the below error while doing
a "nodetool
rebuild -- name_of_existing_data_center" .
[user@machine ~]$ nodetool rebuild DC1
nodetool: Unable to find sufficient sources for streaming range
(-402178150752044282,-396707578307430827] in keyspace system_distributed
See 'nodetool help' or 'nodetool help <command>'.
[user@machine ~]$
user@cqlsh> SELECT * from system_schema.keyspaces where
keyspace_name='system_distributed';
keyspace_name | durable_writes | replication
---------------+----------------+-------------------------------------------------------------------------------------
system_distributed | True | {'class':
'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
(1 rows)
To overcome this I have updated system_distributed keyspace to DC1:3 and
DC2:3 with NetworkTopologyStrategy
C* Version - 3.0.8
Is this a bug that is introduced in 3.0.8 version of cassandra? as I
haven't seen this issue with the older versions?
Re: Rebuild failing while adding new datacenter
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
I have opened https://issues.apache.org/jira/browse/CASSANDRA-12816
On Thu, Oct 20, 2016 at 6:30 PM, Yabin Meng <ya...@gmail.com> wrote:
> Sorry, I'm not aware of it
>
> On Thu, Oct 20, 2016 at 6:00 PM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Thank you Yabin, is there a exisiting JIRA that I can refer to?
>>
>> On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng <ya...@gmail.com> wrote:
>>
>>> I have seen this on other releases, on 2.2.x. The workaround is exactly
>>> like yours, some other system keyspaces also need similar changes.
>>>
>>> I would say this is a benign bug.
>>>
>>> Yabin
>>>
>>> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheemsen@gmail.com> wrote:
>>>
>>>> thanks,
>>>>
>>>> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
>>>> definitely not a firewall issue
>>>>
>>>> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
>>>> pskraju88@gmail.com> wrote:
>>>>
>>>>> we faced a similar issue earlier, but that was more related to
>>>>> firewall rules. The newly added datacenter was not able to communicate with
>>>>> the existing datacenters on the port 7000(inter-node communication). Your's
>>>>> might be a different issue, but just saying.
>>>>>
>>>>>
>>>>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>>>>> jaibheemsen@gmail.com> wrote:
>>>>>
>>>>>> Hello All,
>>>>>>
>>>>>> I have single datacenter with 3 C* nodes and we are trying to expand
>>>>>> the cluster to another region/DC. I am seeing the below error while doing a
>>>>>> "nodetool rebuild -- name_of_existing_data_center" .
>>>>>>
>>>>>> [user@machine ~]$ nodetool rebuild DC1
>>>>>> nodetool: Unable to find sufficient sources for streaming range
>>>>>> (-402178150752044282,-396707578307430827] in keyspace
>>>>>> system_distributed
>>>>>> See 'nodetool help' or 'nodetool help <command>'.
>>>>>> [user@machine ~]$
>>>>>>
>>>>>> user@cqlsh> SELECT * from system_schema.keyspaces where
>>>>>> keyspace_name='system_distributed';
>>>>>>
>>>>>> keyspace_name | durable_writes | replication
>>>>>> ---------------+----------------+---------------------------
>>>>>> ----------------------------------------------------------
>>>>>> system_distributed | True | {'class':
>>>>>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>>>>>> '3'}
>>>>>>
>>>>>> (1 rows)
>>>>>>
>>>>>> To overcome this I have updated system_distributed keyspace to DC1:3
>>>>>> and DC2:3 with NetworkTopologyStrategy
>>>>>>
>>>>>> C* Version - 3.0.8
>>>>>>
>>>>>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>>>>>> haven't seen this issue with the older versions?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Rebuild failing while adding new datacenter
Posted by Yabin Meng <ya...@gmail.com>.
Sorry, I'm not aware of it
On Thu, Oct 20, 2016 at 6:00 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> Thank you Yabin, is there a exisiting JIRA that I can refer to?
>
> On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng <ya...@gmail.com> wrote:
>
>> I have seen this on other releases, on 2.2.x. The workaround is exactly
>> like yours, some other system keyspaces also need similar changes.
>>
>> I would say this is a benign bug.
>>
>> Yabin
>>
>> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> thanks,
>>>
>>> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
>>> definitely not a firewall issue
>>>
>>> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
>>> pskraju88@gmail.com> wrote:
>>>
>>>> we faced a similar issue earlier, but that was more related to firewall
>>>> rules. The newly added datacenter was not able to communicate with the
>>>> existing datacenters on the port 7000(inter-node communication). Your's
>>>> might be a different issue, but just saying.
>>>>
>>>>
>>>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>>>> jaibheemsen@gmail.com> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> I have single datacenter with 3 C* nodes and we are trying to expand
>>>>> the cluster to another region/DC. I am seeing the below error while doing a
>>>>> "nodetool rebuild -- name_of_existing_data_center" .
>>>>>
>>>>> [user@machine ~]$ nodetool rebuild DC1
>>>>> nodetool: Unable to find sufficient sources for streaming range
>>>>> (-402178150752044282,-396707578307430827] in keyspace
>>>>> system_distributed
>>>>> See 'nodetool help' or 'nodetool help <command>'.
>>>>> [user@machine ~]$
>>>>>
>>>>> user@cqlsh> SELECT * from system_schema.keyspaces where
>>>>> keyspace_name='system_distributed';
>>>>>
>>>>> keyspace_name | durable_writes | replication
>>>>> ---------------+----------------+---------------------------
>>>>> ----------------------------------------------------------
>>>>> system_distributed | True | {'class':
>>>>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>>>>> '3'}
>>>>>
>>>>> (1 rows)
>>>>>
>>>>> To overcome this I have updated system_distributed keyspace to DC1:3
>>>>> and DC2:3 with NetworkTopologyStrategy
>>>>>
>>>>> C* Version - 3.0.8
>>>>>
>>>>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>>>>> haven't seen this issue with the older versions?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Rebuild failing while adding new datacenter
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Thank you Yabin, is there a exisiting JIRA that I can refer to?
On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng <ya...@gmail.com> wrote:
> I have seen this on other releases, on 2.2.x. The workaround is exactly
> like yours, some other system keyspaces also need similar changes.
>
> I would say this is a benign bug.
>
> Yabin
>
> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> thanks,
>>
>> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
>> definitely not a firewall issue
>>
>> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
>> pskraju88@gmail.com> wrote:
>>
>>> we faced a similar issue earlier, but that was more related to firewall
>>> rules. The newly added datacenter was not able to communicate with the
>>> existing datacenters on the port 7000(inter-node communication). Your's
>>> might be a different issue, but just saying.
>>>
>>>
>>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheemsen@gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I have single datacenter with 3 C* nodes and we are trying to expand
>>>> the cluster to another region/DC. I am seeing the below error while doing a
>>>> "nodetool rebuild -- name_of_existing_data_center" .
>>>>
>>>> [user@machine ~]$ nodetool rebuild DC1
>>>> nodetool: Unable to find sufficient sources for streaming range
>>>> (-402178150752044282,-396707578307430827] in keyspace
>>>> system_distributed
>>>> See 'nodetool help' or 'nodetool help <command>'.
>>>> [user@machine ~]$
>>>>
>>>> user@cqlsh> SELECT * from system_schema.keyspaces where
>>>> keyspace_name='system_distributed';
>>>>
>>>> keyspace_name | durable_writes | replication
>>>> ---------------+----------------+---------------------------
>>>> ----------------------------------------------------------
>>>> system_distributed | True | {'class':
>>>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>>>> '3'}
>>>>
>>>> (1 rows)
>>>>
>>>> To overcome this I have updated system_distributed keyspace to DC1:3
>>>> and DC2:3 with NetworkTopologyStrategy
>>>>
>>>> C* Version - 3.0.8
>>>>
>>>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>>>> haven't seen this issue with the older versions?
>>>>
>>>
>>>
>>
>
Re: Rebuild failing while adding new datacenter
Posted by Yabin Meng <ya...@gmail.com>.
I have seen this on other releases, on 2.2.x. The workaround is exactly
like yours, some other system keyspaces also need similar changes.
I would say this is a benign bug.
Yabin
On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> thanks,
>
> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
> definitely not a firewall issue
>
> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
> pskraju88@gmail.com> wrote:
>
>> we faced a similar issue earlier, but that was more related to firewall
>> rules. The newly added datacenter was not able to communicate with the
>> existing datacenters on the port 7000(inter-node communication). Your's
>> might be a different issue, but just saying.
>>
>>
>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I have single datacenter with 3 C* nodes and we are trying to expand the
>>> cluster to another region/DC. I am seeing the below error while doing a "nodetool
>>> rebuild -- name_of_existing_data_center" .
>>>
>>> [user@machine ~]$ nodetool rebuild DC1
>>> nodetool: Unable to find sufficient sources for streaming range
>>> (-402178150752044282,-396707578307430827] in keyspace system_distributed
>>> See 'nodetool help' or 'nodetool help <command>'.
>>> [user@machine ~]$
>>>
>>> user@cqlsh> SELECT * from system_schema.keyspaces where
>>> keyspace_name='system_distributed';
>>>
>>> keyspace_name | durable_writes | replication
>>> ---------------+----------------+---------------------------
>>> ----------------------------------------------------------
>>> system_distributed | True | {'class':
>>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>>> '3'}
>>>
>>> (1 rows)
>>>
>>> To overcome this I have updated system_distributed keyspace to DC1:3 and
>>> DC2:3 with NetworkTopologyStrategy
>>>
>>> C* Version - 3.0.8
>>>
>>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>>> haven't seen this issue with the older versions?
>>>
>>
>>
>
Re: Rebuild failing while adding new datacenter
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
thanks,
This always works on 2.1.13 and 2.1.16 version but not on 3.0.8. definitely
not a firewall issue
On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
pskraju88@gmail.com> wrote:
> we faced a similar issue earlier, but that was more related to firewall
> rules. The newly added datacenter was not able to communicate with the
> existing datacenters on the port 7000(inter-node communication). Your's
> might be a different issue, but just saying.
>
>
> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Hello All,
>>
>> I have single datacenter with 3 C* nodes and we are trying to expand the
>> cluster to another region/DC. I am seeing the below error while doing a "nodetool
>> rebuild -- name_of_existing_data_center" .
>>
>> [user@machine ~]$ nodetool rebuild DC1
>> nodetool: Unable to find sufficient sources for streaming range
>> (-402178150752044282,-396707578307430827] in keyspace system_distributed
>> See 'nodetool help' or 'nodetool help <command>'.
>> [user@machine ~]$
>>
>> user@cqlsh> SELECT * from system_schema.keyspaces where
>> keyspace_name='system_distributed';
>>
>> keyspace_name | durable_writes | replication
>> ---------------+----------------+---------------------------
>> ----------------------------------------------------------
>> system_distributed | True | {'class':
>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
>>
>> (1 rows)
>>
>> To overcome this I have updated system_distributed keyspace to DC1:3 and
>> DC2:3 with NetworkTopologyStrategy
>>
>> C* Version - 3.0.8
>>
>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>> haven't seen this issue with the older versions?
>>
>
>
Re: Rebuild failing while adding new datacenter
Posted by sai krishnam raju potturi <ps...@gmail.com>.
we faced a similar issue earlier, but that was more related to firewall
rules. The newly added datacenter was not able to communicate with the
existing datacenters on the port 7000(inter-node communication). Your's
might be a different issue, but just saying.
On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> Hello All,
>
> I have single datacenter with 3 C* nodes and we are trying to expand the
> cluster to another region/DC. I am seeing the below error while doing a "nodetool
> rebuild -- name_of_existing_data_center" .
>
> [user@machine ~]$ nodetool rebuild DC1
> nodetool: Unable to find sufficient sources for streaming range
> (-402178150752044282,-396707578307430827] in keyspace system_distributed
> See 'nodetool help' or 'nodetool help <command>'.
> [user@machine ~]$
>
> user@cqlsh> SELECT * from system_schema.keyspaces where
> keyspace_name='system_distributed';
>
> keyspace_name | durable_writes | replication
> ---------------+----------------+---------------------------
> ----------------------------------------------------------
> system_distributed | True | {'class':
> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
>
> (1 rows)
>
> To overcome this I have updated system_distributed keyspace to DC1:3 and
> DC2:3 with NetworkTopologyStrategy
>
> C* Version - 3.0.8
>
> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
> haven't seen this issue with the older versions?
>