You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/08/04 12:52:27 UTC
RuntimeException: Cannot service reads while bootstrapping!
Is this a known issue?
Running 0.6.2 I moved a node to different token and eventually saw errors in
the log.
ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.RuntimeException: Cannot service reads while bootstrapping!
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
(line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
java.lang.RuntimeException: Cannot service reads while bootstrapping!
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
... many more of those and then...
INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
StorageService.java (line 181) Bootstrap/move completed! Now serving reads.
The move ended up "ok" but during the operation the log was filled with
those errors and at the end of it the ring state was inconsistent.
If I ask the moved node where it is in the ring it tells me something but
other nodes tell something else...
(ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.254.58 -p 9004 ring
Address Status Load Range
Ring
170141183460469231731687303715884105727
192.168.252.88Up 5.7 GB
14131484407726020523932116250949797205 |<--|
192.168.252.124Up 2.44 GB
56713727820156410577229101238628035242 | ^
192.168.254.58Up 8.13 GB *
113427455640312821154458202477256070484* v |
192.168.254.57Up 6.52 GB
113427455640312821154458202477256070485 | ^
192.168.252.125Up 6.52 GB
141784319550391026443072753096570088105 v |
192.168.254.59Up 1.63 GB
170141183460469231731687303715884105727 |-->|
(ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.252.124 -p 9004 ring
Address Status Load Range
Ring
170141183460469231731687303715884105727
192.168.252.88Up 5.7 GB
14131484407726020523932116250949797205 |<--|
192.168.252.124Up 2.46 GB
56713727820156410577229101238628035242 | ^
192.168.254.57Up 6.52 GB
113427455640312821154458202477256070485 v |
192.168.252.125Up 6.52 GB
141784319550391026443072753096570088105 | ^
192.168.254.58Up 1.63 GB *
141784319550391026443072753096570088106* v |
192.168.254.59Up 1.63 GB
170141183460469231731687303715884105727 |-->|
Restarting the moved node fixes the ring view by other hosts.
Re: RuntimeException: Cannot service reads while bootstrapping!
Posted by Ran Tavory <ra...@gmail.com>.
reported https://issues.apache.org/jira/browse/CASSANDRA-1359
On Wed, Aug 4, 2010 at 9:51 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> That's definitely a bug.
>
> On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <ra...@gmail.com> wrote:
> > ok, so I don't send writes to bootstrapping or decommissioned nodes,
> that's
> > cool, but what about the inconsistent ring view after nodetool move,
> isn't
> > this strange?
> > After the move, the moved node has the correct view of the ring but all
> > other nodes have the old view. I waited a few minutes after the log said
> > that Bootstrap/move completed! Now serving reads but this didn't help,
> view
> > was still inconsistent. Only restarting the moved node helped other nodes
> > realize the change.
> > On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> Don't point clients at nodes that aren't part of the ring. Cassandra
> >> rejecting requests when you do is a "feature."
> >>
> >> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <ra...@gmail.com> wrote:
> >> > Is this a known issue?
> >> > Running 0.6.2 I moved a node to different token and eventually saw
> >> > errors in
> >> > the log.
> >> >
> >> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> >> > DebuggableThreadPoolExecutor.java (line 101) Error in
> ThreadPoolExecutor
> >> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >> > at
> >> >
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >> > at
> >> >
> >> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >> > at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > at java.lang.Thread.run(Thread.java:619)
> >> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
> >> > CassandraDaemon.java
> >> > (line 82) Fatal exception in thread
> Thread[ROW-READ-STAGE:116805,5,main]
> >> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >> > at
> >> >
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >> > at
> >> >
> >> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >> > at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > at java.lang.Thread.run(Thread.java:619)
> >> > ... many more of those and then...
> >> > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> >> > StorageService.java (line 181) Bootstrap/move completed! Now serving
> >> > reads.
> >> >
> >> > The move ended up "ok" but during the operation the log was filled
> with
> >> > those errors and at the end of it the ring state was inconsistent.
> >> > If I ask the moved node where it is in the ring it tells me something
> >> > but
> >> > other nodes tell something else...
> >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> >> > 192.168.254.58 -p 9004 ring
> >> > Address Status Load Range
> >> > Ring
> >> >
> >> > 170141183460469231731687303715884105727
> >> > 192.168.252.88Up 5.7 GB
> >> > 14131484407726020523932116250949797205 |<--|
> >> > 192.168.252.124Up 2.44 GB
> >> > 56713727820156410577229101238628035242 | ^
> >> > 192.168.254.58Up 8.13 GB
> >> > 113427455640312821154458202477256070484 v |
> >> > 192.168.254.57Up 6.52 GB
> >> > 113427455640312821154458202477256070485 | ^
> >> > 192.168.252.125Up 6.52 GB
> >> > 141784319550391026443072753096570088105 v |
> >> > 192.168.254.59Up 1.63 GB
> >> > 170141183460469231731687303715884105727 |-->|
> >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> >> > -h 192.168.252.124 -p 9004 ring
> >> > Address Status Load Range
> >> > Ring
> >> >
> >> > 170141183460469231731687303715884105727
> >> > 192.168.252.88Up 5.7 GB
> >> > 14131484407726020523932116250949797205 |<--|
> >> > 192.168.252.124Up 2.46 GB
> >> > 56713727820156410577229101238628035242 | ^
> >> > 192.168.254.57Up 6.52 GB
> >> > 113427455640312821154458202477256070485 v |
> >> > 192.168.252.125Up 6.52 GB
> >> > 141784319550391026443072753096570088105 | ^
> >> > 192.168.254.58Up 1.63 GB
> >> > 141784319550391026443072753096570088106 v |
> >> > 192.168.254.59Up 1.63 GB
> >> > 170141183460469231731687303715884105727 |-->|
> >> > Restarting the moved node fixes the ring view by other hosts.
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of Riptano, the source for professional Cassandra support
> >> http://riptano.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: RuntimeException: Cannot service reads while bootstrapping!
Posted by Jonathan Ellis <jb...@gmail.com>.
That's definitely a bug.
On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <ra...@gmail.com> wrote:
> ok, so I don't send writes to bootstrapping or decommissioned nodes, that's
> cool, but what about the inconsistent ring view after nodetool move, isn't
> this strange?
> After the move, the moved node has the correct view of the ring but all
> other nodes have the old view. I waited a few minutes after the log said
> that Bootstrap/move completed! Now serving reads but this didn't help, view
> was still inconsistent. Only restarting the moved node helped other nodes
> realize the change.
> On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> Don't point clients at nodes that aren't part of the ring. Cassandra
>> rejecting requests when you do is a "feature."
>>
>> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <ra...@gmail.com> wrote:
>> > Is this a known issue?
>> > Running 0.6.2 I moved a node to different token and eventually saw
>> > errors in
>> > the log.
>> >
>> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
>> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> > at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>> > at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:619)
>> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
>> > CassandraDaemon.java
>> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> > at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>> > at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:619)
>> > ... many more of those and then...
>> > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
>> > StorageService.java (line 181) Bootstrap/move completed! Now serving
>> > reads.
>> >
>> > The move ended up "ok" but during the operation the log was filled with
>> > those errors and at the end of it the ring state was inconsistent.
>> > If I ask the moved node where it is in the ring it tells me something
>> > but
>> > other nodes tell something else...
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
>> > 192.168.254.58 -p 9004 ring
>> > Address Status Load Range
>> > Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up 5.7 GB
>> > 14131484407726020523932116250949797205 |<--|
>> > 192.168.252.124Up 2.44 GB
>> > 56713727820156410577229101238628035242 | ^
>> > 192.168.254.58Up 8.13 GB
>> > 113427455640312821154458202477256070484 v |
>> > 192.168.254.57Up 6.52 GB
>> > 113427455640312821154458202477256070485 | ^
>> > 192.168.252.125Up 6.52 GB
>> > 141784319550391026443072753096570088105 v |
>> > 192.168.254.59Up 1.63 GB
>> > 170141183460469231731687303715884105727 |-->|
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
>> > -h 192.168.252.124 -p 9004 ring
>> > Address Status Load Range
>> > Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up 5.7 GB
>> > 14131484407726020523932116250949797205 |<--|
>> > 192.168.252.124Up 2.46 GB
>> > 56713727820156410577229101238628035242 | ^
>> > 192.168.254.57Up 6.52 GB
>> > 113427455640312821154458202477256070485 v |
>> > 192.168.252.125Up 6.52 GB
>> > 141784319550391026443072753096570088105 | ^
>> > 192.168.254.58Up 1.63 GB
>> > 141784319550391026443072753096570088106 v |
>> > 192.168.254.59Up 1.63 GB
>> > 170141183460469231731687303715884105727 |-->|
>> > Restarting the moved node fixes the ring view by other hosts.
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: RuntimeException: Cannot service reads while bootstrapping!
Posted by Ran Tavory <ra...@gmail.com>.
ok, so I don't send writes to bootstrapping or decommissioned nodes, that's
cool, but what about the inconsistent ring view after nodetool move, isn't
this strange?
After the move, the moved node has the correct view of the ring but all
other nodes have the old view. I waited a few minutes after the log said
that Bootstrap/move completed! Now serving reads but this didn't help, view
was still inconsistent. Only restarting the moved node helped other nodes
realize the change.
On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Don't point clients at nodes that aren't part of the ring. Cassandra
> rejecting requests when you do is a "feature."
>
> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <ra...@gmail.com> wrote:
> > Is this a known issue?
> > Running 0.6.2 I moved a node to different token and eventually saw errors
> in
> > the log.
> >
> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> > at
> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> > at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
> CassandraDaemon.java
> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> > at
> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> > at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > ... many more of those and then...
> > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> > StorageService.java (line 181) Bootstrap/move completed! Now serving
> reads.
> >
> > The move ended up "ok" but during the operation the log was filled with
> > those errors and at the end of it the ring state was inconsistent.
> > If I ask the moved node where it is in the ring it tells me something but
> > other nodes tell something else...
> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> > 192.168.254.58 -p 9004 ring
> > Address Status Load Range
> > Ring
> >
> > 170141183460469231731687303715884105727
> > 192.168.252.88Up 5.7 GB
> > 14131484407726020523932116250949797205 |<--|
> > 192.168.252.124Up 2.44 GB
> > 56713727820156410577229101238628035242 | ^
> > 192.168.254.58Up 8.13 GB
> > 113427455640312821154458202477256070484 v |
> > 192.168.254.57Up 6.52 GB
> > 113427455640312821154458202477256070485 | ^
> > 192.168.252.125Up 6.52 GB
> > 141784319550391026443072753096570088105 v |
> > 192.168.254.59Up 1.63 GB
> > 170141183460469231731687303715884105727 |-->|
> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> > -h 192.168.252.124 -p 9004 ring
> > Address Status Load Range
> > Ring
> >
> > 170141183460469231731687303715884105727
> > 192.168.252.88Up 5.7 GB
> > 14131484407726020523932116250949797205 |<--|
> > 192.168.252.124Up 2.46 GB
> > 56713727820156410577229101238628035242 | ^
> > 192.168.254.57Up 6.52 GB
> > 113427455640312821154458202477256070485 v |
> > 192.168.252.125Up 6.52 GB
> > 141784319550391026443072753096570088105 | ^
> > 192.168.254.58Up 1.63 GB
> > 141784319550391026443072753096570088106 v |
> > 192.168.254.59Up 1.63 GB
> > 170141183460469231731687303715884105727 |-->|
> > Restarting the moved node fixes the ring view by other hosts.
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: RuntimeException: Cannot service reads while bootstrapping!
Posted by Jonathan Ellis <jb...@gmail.com>.
Don't point clients at nodes that aren't part of the ring. Cassandra
rejecting requests when you do is a "feature."
On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <ra...@gmail.com> wrote:
> Is this a known issue?
> Running 0.6.2 I moved a node to different token and eventually saw errors in
> the log.
>
> ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
> (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ... many more of those and then...
> INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> StorageService.java (line 181) Bootstrap/move completed! Now serving reads.
>
> The move ended up "ok" but during the operation the log was filled with
> those errors and at the end of it the ring state was inconsistent.
> If I ask the moved node where it is in the ring it tells me something but
> other nodes tell something else...
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> 192.168.254.58 -p 9004 ring
> Address Status Load Range
> Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up 5.7 GB
> 14131484407726020523932116250949797205 |<--|
> 192.168.252.124Up 2.44 GB
> 56713727820156410577229101238628035242 | ^
> 192.168.254.58Up 8.13 GB
> 113427455640312821154458202477256070484 v |
> 192.168.254.57Up 6.52 GB
> 113427455640312821154458202477256070485 | ^
> 192.168.252.125Up 6.52 GB
> 141784319550391026443072753096570088105 v |
> 192.168.254.59Up 1.63 GB
> 170141183460469231731687303715884105727 |-->|
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> -h 192.168.252.124 -p 9004 ring
> Address Status Load Range
> Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up 5.7 GB
> 14131484407726020523932116250949797205 |<--|
> 192.168.252.124Up 2.46 GB
> 56713727820156410577229101238628035242 | ^
> 192.168.254.57Up 6.52 GB
> 113427455640312821154458202477256070485 v |
> 192.168.252.125Up 6.52 GB
> 141784319550391026443072753096570088105 | ^
> 192.168.254.58Up 1.63 GB
> 141784319550391026443072753096570088106 v |
> 192.168.254.59Up 1.63 GB
> 170141183460469231731687303715884105727 |-->|
> Restarting the moved node fixes the ring view by other hosts.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com