You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rock Zhang <ro...@alohar.com> on 2015/09/15 09:14:08 UTC
Cassandra adding node issue (no UJ status)
Hi All,
Now I got a problem everything when add new node, the data is not balance
again, just add it as new node.
Originally I saw the UJ status, anybody experienced this kind of issue ? I
don't know what changed .
ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.31.16.191 742.15 GB 256 21.1%
08638815-b721-46c4-b77c-af08285226db RAC1
UN 172.31.30.145 774.42 GB 256 21.2%
bde31dd9-ff1d-4f2f-b28d-fe54d0531c51 RAC1
UN 172.31.6.79 592.9 GB 256 19.8%
15795fca-5425-41cd-909c-c1756715442a RAC2
UN 172.31.27.186 674.42 GB 256 18.4%
9685a476-1da7-4c6f-819e-dd4483e3345e RAC1
UN 172.31.7.31 642.47 GB 256 19.9%
f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b RAC1
*UN 172.31.15.242 37.4 MB 256 19.8%
c3eff010-9904-49a0-83cd-258dc5a98525 RAC1*
UN 172.31.24.32 780.59 GB 256 20.1%
ffa58bd1-3188-440d-94c9-97166ee4b735 RAC1
*UN 172.31.3.40 80.75 GB 256 18.9%
01ce3f96-ebc0-4128-9ec3-ddd1a9845d51 RAC1*
UN 172.31.31.238 756.59 GB 256 19.9%
82d34a3b-4f12-4874-816c-7d89a0535577 RAC1
UN 172.31.31.99 583.68 GB 256 20.8%
2b10194f-23d2-4bdc-bcfa-7961a149cd11 RAC2
Thanks
Rock
Re: Cassandra adding node issue (no UJ status)
Posted by Rock Zhang <ro...@alohar.com>.
Thank you, Mark.
On Tue, Sep 15, 2015 at 5:44 AM, Mark Greene <gr...@gmail.com> wrote:
> Hey Rock,
>
> I've seen this occur as well. I've come to learn that in some cases, like
> a network blip, the join can fail. There is usually something in the log to
> the effect of "Stream failed"
>
> When I encounter this issue, I make an attempt to bootstrap the new node
> again. If that doesn't help, I run a repair on the new node.
>
>
> On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang <ro...@alohar.com> wrote:
>
>> Hi All,
>>
>>
>> Now I got a problem everything when add new node, the data is not balance
>> again, just add it as new node.
>>
>>
>> Originally I saw the UJ status, anybody experienced this kind of issue ?
>> I don't know what changed .
>>
>>
>> ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
>>
>> Datacenter: DC1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 172.31.16.191 742.15 GB 256 21.1%
>> 08638815-b721-46c4-b77c-af08285226db RAC1
>>
>> UN 172.31.30.145 774.42 GB 256 21.2%
>> bde31dd9-ff1d-4f2f-b28d-fe54d0531c51 RAC1
>>
>> UN 172.31.6.79 592.9 GB 256 19.8%
>> 15795fca-5425-41cd-909c-c1756715442a RAC2
>>
>> UN 172.31.27.186 674.42 GB 256 18.4%
>> 9685a476-1da7-4c6f-819e-dd4483e3345e RAC1
>>
>> UN 172.31.7.31 642.47 GB 256 19.9%
>> f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b RAC1
>>
>> *UN 172.31.15.242 37.4 MB 256 19.8%
>> c3eff010-9904-49a0-83cd-258dc5a98525 RAC1*
>>
>> UN 172.31.24.32 780.59 GB 256 20.1%
>> ffa58bd1-3188-440d-94c9-97166ee4b735 RAC1
>>
>> *UN 172.31.3.40 80.75 GB 256 18.9%
>> 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51 RAC1*
>>
>> UN 172.31.31.238 756.59 GB 256 19.9%
>> 82d34a3b-4f12-4874-816c-7d89a0535577 RAC1
>>
>> UN 172.31.31.99 583.68 GB 256 20.8%
>> 2b10194f-23d2-4bdc-bcfa-7961a149cd11 RAC2
>>
>>
>> Thanks
>>
>> Rock
>>
>
>
Re: Cassandra adding node issue (no UJ status)
Posted by Sebastian Estevez <se...@datastax.com>.
Check https://issues.apache.org/jira/browse/CASSANDRA-8611
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Tue, Sep 15, 2015 at 8:44 AM, Mark Greene <gr...@gmail.com> wrote:
> Hey Rock,
>
> I've seen this occur as well. I've come to learn that in some cases, like
> a network blip, the join can fail. There is usually something in the log to
> the effect of "Stream failed"
>
> When I encounter this issue, I make an attempt to bootstrap the new node
> again. If that doesn't help, I run a repair on the new node.
>
>
> On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang <ro...@alohar.com> wrote:
>
>> Hi All,
>>
>>
>> Now I got a problem everything when add new node, the data is not balance
>> again, just add it as new node.
>>
>>
>> Originally I saw the UJ status, anybody experienced this kind of issue ?
>> I don't know what changed .
>>
>>
>> ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
>>
>> Datacenter: DC1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 172.31.16.191 742.15 GB 256 21.1%
>> 08638815-b721-46c4-b77c-af08285226db RAC1
>>
>> UN 172.31.30.145 774.42 GB 256 21.2%
>> bde31dd9-ff1d-4f2f-b28d-fe54d0531c51 RAC1
>>
>> UN 172.31.6.79 592.9 GB 256 19.8%
>> 15795fca-5425-41cd-909c-c1756715442a RAC2
>>
>> UN 172.31.27.186 674.42 GB 256 18.4%
>> 9685a476-1da7-4c6f-819e-dd4483e3345e RAC1
>>
>> UN 172.31.7.31 642.47 GB 256 19.9%
>> f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b RAC1
>>
>> *UN 172.31.15.242 37.4 MB 256 19.8%
>> c3eff010-9904-49a0-83cd-258dc5a98525 RAC1*
>>
>> UN 172.31.24.32 780.59 GB 256 20.1%
>> ffa58bd1-3188-440d-94c9-97166ee4b735 RAC1
>>
>> *UN 172.31.3.40 80.75 GB 256 18.9%
>> 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51 RAC1*
>>
>> UN 172.31.31.238 756.59 GB 256 19.9%
>> 82d34a3b-4f12-4874-816c-7d89a0535577 RAC1
>>
>> UN 172.31.31.99 583.68 GB 256 20.8%
>> 2b10194f-23d2-4bdc-bcfa-7961a149cd11 RAC2
>>
>>
>> Thanks
>>
>> Rock
>>
>
>
Re: Cassandra adding node issue (no UJ status)
Posted by Mark Greene <gr...@gmail.com>.
Hey Rock,
I've seen this occur as well. I've come to learn that in some cases, like a
network blip, the join can fail. There is usually something in the log to
the effect of "Stream failed"
When I encounter this issue, I make an attempt to bootstrap the new node
again. If that doesn't help, I run a repair on the new node.
On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang <ro...@alohar.com> wrote:
> Hi All,
>
>
> Now I got a problem everything when add new node, the data is not balance
> again, just add it as new node.
>
>
> Originally I saw the UJ status, anybody experienced this kind of issue ? I
> don't know what changed .
>
>
> ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
>
> Datacenter: DC1
>
> ===============
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns (effective) Host ID
> Rack
>
> UN 172.31.16.191 742.15 GB 256 21.1%
> 08638815-b721-46c4-b77c-af08285226db RAC1
>
> UN 172.31.30.145 774.42 GB 256 21.2%
> bde31dd9-ff1d-4f2f-b28d-fe54d0531c51 RAC1
>
> UN 172.31.6.79 592.9 GB 256 19.8%
> 15795fca-5425-41cd-909c-c1756715442a RAC2
>
> UN 172.31.27.186 674.42 GB 256 18.4%
> 9685a476-1da7-4c6f-819e-dd4483e3345e RAC1
>
> UN 172.31.7.31 642.47 GB 256 19.9%
> f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b RAC1
>
> *UN 172.31.15.242 37.4 MB 256 19.8%
> c3eff010-9904-49a0-83cd-258dc5a98525 RAC1*
>
> UN 172.31.24.32 780.59 GB 256 20.1%
> ffa58bd1-3188-440d-94c9-97166ee4b735 RAC1
>
> *UN 172.31.3.40 80.75 GB 256 18.9%
> 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51 RAC1*
>
> UN 172.31.31.238 756.59 GB 256 19.9%
> 82d34a3b-4f12-4874-816c-7d89a0535577 RAC1
>
> UN 172.31.31.99 583.68 GB 256 20.8%
> 2b10194f-23d2-4bdc-bcfa-7961a149cd11 RAC2
>
>
> Thanks
>
> Rock
>