You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/05/26 11:54:35 UTC

nodetool move looks stuck

I ran nodetool move on one of the nodes and it seems stuck for a few hours
now.

I've been able to run it successfully in the past, but this time it looks
stuck.

Streams shows as if there's work in progress, but the same files have been
at the same position for a few hours.
I've also checked the compaction manager through jmx and it's not compacting
anything.

Anything I can do to continue the move operation? Is this a bug?
thanks


$ nodetool -h 192.168.254.58 -p 9004 streams
Mode: Normal
 Nothing streaming to /192.168.252.124
Streaming to: /192.168.254.57
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Filter.db
0/45587965
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Data.db
0/18369406636
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.db
0/6163411
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db
0/85645
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.db
0/37985032
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db
0/77307418
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db
0/1436005
   /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db
0/704736670
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db 0/230280
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db 0/20605
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db
0/202808000
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db 0/23912
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db 0/20605
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db 0/21434973
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db 0/965028
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db 0/20605
   /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db
0/865380205
Not receiving any streams.

$ nodetool -h 192.168.254.57 -p 9004 streams
Mode: Normal
 Nothing streaming to /192.168.254.58
Streaming from: /192.168.254.58
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db
0/18369406636
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db 0/6163411
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db 0/85645
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db 0/37985032
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005
   outbrain_kvdb:
/outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Index.db
0/230280
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Filter.db
0/20605
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db
0/202808000
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Index.db
0/23912
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Filter.db
0/20605
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db
0/21434973
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Index.db
0/965028
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Filter.db
0/20605
   Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db
0/865380205

Re: nodetool move looks stuck

Posted by Ran Tavory <ra...@gmail.com>.

I don't see any errors in neither the receiver nor sender. The cluster
itself looks healthy (e.g. no random networks disconnections or anything
like that as described in 1019).

I'll restart the moved node, hope it'll continue its more afterwards but it
does look like a bug. Is there something I can do to help debug it?
I'll attach a thread dump here just in case before I restart it



On Wed, May 26, 2010 at 9:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:

> Are there any exceptions in the log like the one in
> https://issues.apache.org/jira/browse/CASSANDRA-1019 ?
>
> If so you'll need to restart the moving node and try again.
>
> On Wed, May 26, 2010 at 3:54 AM, Ran Tavory <ra...@gmail.com> wrote:
> > I ran nodetool move on one of the nodes and it seems stuck for a few
> hours
> > now.
> > I've been able to run it successfully in the past, but this time it looks
> > stuck.
> > Streams shows as if there's work in progress, but the same files have
> been
> > at the same position for a few hours.
> > I've also checked the compaction manager through jmx and it's not
> compacting
> > anything.
> > Anything I can do to continue the move operation? Is this a bug?
> > thanks
> >
> > $ nodetool -h 192.168.254.58 -p 9004 streams
> > Mode: Normal
> >  Nothing streaming to /192.168.252.124
> > Streaming to: /192.168.254.57
> >
> /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Filter.db
> > 0/45587965
> >
> /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Data.db
> > 0/18369406636
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.db
> > 0/6163411
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db
> > 0/85645
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.db
> > 0/37985032
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db
> > 0/77307418
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db
> > 0/1436005
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db
> > 0/704736670
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db
> 0/230280
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db
> > 0/202808000
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db
> 0/23912
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db
> 0/21434973
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db
> 0/965028
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db
> > 0/865380205
> > Not receiving any streams.
> > $ nodetool -h 192.168.254.57 -p 9004 streams
> > Mode: Normal
> >  Nothing streaming to /192.168.254.58
> > Streaming from: /192.168.254.58
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db
> > 0/18369406636
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db
> 0/6163411
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db
> 0/85645
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db
> 0/37985032
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Index.db
> > 0/230280
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db
> > 0/202808000
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Index.db
> > 0/23912
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db
> > 0/21434973
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Index.db
> > 0/965028
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db
> > 0/865380205
> >
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: nodetool move looks stuck

Posted by Jonathan Ellis <jb...@gmail.com>.

Are there any exceptions in the log like the one in
https://issues.apache.org/jira/browse/CASSANDRA-1019 ?

If so you'll need to restart the moving node and try again.

On Wed, May 26, 2010 at 3:54 AM, Ran Tavory <ra...@gmail.com> wrote:
> I ran nodetool move on one of the nodes and it seems stuck for a few hours
> now.
> I've been able to run it successfully in the past, but this time it looks
> stuck.
> Streams shows as if there's work in progress, but the same files have been
> at the same position for a few hours.
> I've also checked the compaction manager through jmx and it's not compacting
> anything.
> Anything I can do to continue the move operation? Is this a bug?
> thanks
>
> $ nodetool -h 192.168.254.58 -p 9004 streams
> Mode: Normal
>  Nothing streaming to /192.168.252.124
> Streaming to: /192.168.254.57
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Filter.db
> 0/45587965
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Data.db
> 0/18369406636
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.db
> 0/6163411
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db
> 0/85645
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.db
> 0/37985032
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db
> 0/77307418
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db
> 0/1436005
>    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db
> 0/704736670
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db 0/230280
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db 0/20605
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db
> 0/202808000
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db 0/23912
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db 0/20605
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db 0/21434973
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db 0/965028
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db 0/20605
>    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db
> 0/865380205
> Not receiving any streams.
> $ nodetool -h 192.168.254.57 -p 9004 streams
> Mode: Normal
>  Nothing streaming to /192.168.254.58
> Streaming from: /192.168.254.58
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db
> 0/18369406636
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db 0/6163411
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db 0/85645
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db 0/37985032
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005
>    outbrain_kvdb:
> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Index.db
> 0/230280
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Filter.db
> 0/20605
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db
> 0/202808000
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Index.db
> 0/23912
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Filter.db
> 0/20605
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db
> 0/21434973
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Index.db
> 0/965028
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Filter.db
> 0/20605
>    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db
> 0/865380205
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com