You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Daning Wang <da...@netseer.com> on 2012/03/21 18:24:21 UTC
Cassandra Exception
Hi All,
We got lots of Exception in the log, and later the server crashed. any idea
what is happening and how to fix it?
ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:4,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:2,5,main]
java.io.IOError:
org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
cfId=-387130991
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at
org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
Couldn't find cfId=-387130991
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
The is the exception before server crashes.
ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[ReadRepairStage:299,5,main]
java.lang.RuntimeException:
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.RejectedExecutionException:
ThreadPoolExecutor has shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
at
org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
at
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
at
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
Thank you in advance,
Daning
Re: Cassandra Exception
Posted by Daning Wang <da...@netseer.com>.
We upgraded to 1.0.8, and looks the problem is gone.
Thanks for your help,
Daning
On Sun, Mar 25, 2012 at 9:54 AM, aaron morton <aa...@thelastpickle.com>wrote:
> Can you go to those nodes and run describe cluster ? Also check the logs
> on the machines that are marked as UNREACHABLE .
>
> A node will be marked as UNREACHABLE if it is DOWN or if it did not
> respond in time.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 23/03/2012, at 11:29 AM, Daning Wang wrote:
>
> Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE",
> but nodetool ring is fine. it is pretty busy cluster, read 3K/sec
>
> $ cassandra-cli -h localhost -u root -pw cassy
> Connected to: "Production Cluster" on localhost/9160
> Welcome to the Cassandra CLI.
>
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
>
> [root@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48,
> 10.240.113.171]
> e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200,
> 10.6.115.239]
> [root@unknown]
>
> $ nodetool -h localhost ring
> Address DC Rack Status State Load
> Owns Token
>
> 141784319550391026443072753096570088105
> 10.28.78.123 datacenter1 rack1 Up Normal 5.46 GB
> 16.67% 0
> 10.120.205.48 datacenter1 rack1 Up Normal 5.49 GB
> 16.67% 28356863910078205288614550619314017621
> 10.6.115.239 datacenter1 rack1 Up Normal 5.53 GB
> 16.67% 56713727820156410577229101238628035242
> 10.28.20.200 datacenter1 rack1 Up Normal 5.51 GB
> 16.67% 85070591730234615865843651857942052863
> 10.123.83.114 datacenter1 rack1 Up Normal 5.49 GB
> 16.67% 113427455640312821154458202477256070484
> 10.240.113.171 datacenter1 rack1 Up Normal 5.43 GB
> 16.67% 141784319550391026443072753096570088105
>
>
> Daning
>
>
> On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> java.io.IOError:
>>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>>> cfId=-387130991
>>>
>> Schema may have diverged between nodes.
>>
>> use cassandra-cli and run describe cluster; to see how many schema
>> versions you have.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>>
>> and we are on 0.8.6.
>>
>>
>>
>> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <da...@netseer.com> wrote:
>>
>>> Hi All,
>>>
>>>
>>> We got lots of Exception in the log, and later the server crashed. any
>>> idea what is happening and how to fix it?
>>>
>>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[RequestResponseStage:4,5,main]
>>> java.io.IOError: java.io.EOFException
>>> at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>> at
>>> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>>> at
>>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>> at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.io.EOFException
>>> at java.io.DataInputStream.readFully(DataInputStream.java:197)
>>> at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>>> at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>> at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>>> ... 6 more
>>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[RequestResponseStage:2,5,main]
>>> java.io.IOError:
>>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>>> cfId=-387130991
>>> at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>> at
>>> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>>> at
>>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>> at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
>>> Couldn't find cfId=-387130991
>>> at
>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>>> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>>> at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>>> at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>> at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>>
>>>
>>>
>>> The is the exception before server crashes.
>>>
>>>
>>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[ReadRepairStage:299,5,main]
>>> java.lang.RuntimeException:
>>> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
>>> shut down
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.util.concurrent.RejectedExecutionException:
>>> ThreadPoolExecutor has shut down
>>> at
>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>> at
>>> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>>> at
>>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>>> at
>>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>>> at
>>> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>>> at
>>> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>>> at
>>> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> ... 3 more
>>>
>>> Thank you in advance,
>>>
>>> Daning
>>>
>>
>>
>>
>
>
Re: Cassandra Exception
Posted by aaron morton <aa...@thelastpickle.com>.
Can you go to those nodes and run describe cluster ? Also check the logs on the machines that are marked as UNREACHABLE .
A node will be marked as UNREACHABLE if it is DOWN or if it did not respond in time.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 23/03/2012, at 11:29 AM, Daning Wang wrote:
> Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE", but nodetool ring is fine. it is pretty busy cluster, read 3K/sec
>
> $ cassandra-cli -h localhost -u root -pw cassy
> Connected to: "Production Cluster" on localhost/9160
> Welcome to the Cassandra CLI.
>
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
>
> [root@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48, 10.240.113.171]
> e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200, 10.6.115.239]
> [root@unknown]
>
> $ nodetool -h localhost ring
> Address DC Rack Status State Load Owns Token
> 141784319550391026443072753096570088105
> 10.28.78.123 datacenter1 rack1 Up Normal 5.46 GB 16.67% 0
> 10.120.205.48 datacenter1 rack1 Up Normal 5.49 GB 16.67% 28356863910078205288614550619314017621
> 10.6.115.239 datacenter1 rack1 Up Normal 5.53 GB 16.67% 56713727820156410577229101238628035242
> 10.28.20.200 datacenter1 rack1 Up Normal 5.51 GB 16.67% 85070591730234615865843651857942052863
> 10.123.83.114 datacenter1 rack1 Up Normal 5.49 GB 16.67% 113427455640312821154458202477256070484
> 10.240.113.171 datacenter1 rack1 Up Normal 5.43 GB 16.67% 141784319550391026443072753096570088105
>
>
> Daning
>
>
> On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
> Schema may have diverged between nodes.
>
> use cassandra-cli and run describe cluster; to see how many schema versions you have.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>
>> and we are on 0.8.6.
>>
>>
>>
>> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <da...@netseer.com> wrote:
>> Hi All,
>>
>>
>> We got lots of Exception in the log, and later the server crashed. any idea what is happening and how to fix it?
>>
>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[RequestResponseStage:4,5,main]
>> java.io.IOError: java.io.EOFException
>> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>> ... 6 more
>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[RequestResponseStage:2,5,main]
>> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
>> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
>> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>
>>
>>
>> The is the exception before server crashes.
>>
>>
>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadRepairStage:299,5,main]
>> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
>> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>> at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>> at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>> at org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>> Thank you in advance,
>>
>> Daning
>>
>
>
Re: Cassandra Exception
Posted by Daning Wang <da...@netseer.com>.
Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE",
but nodetool ring is fine. it is pretty busy cluster, read 3K/sec
$ cassandra-cli -h localhost -u root -pw cassy
Connected to: "Production Cluster" on localhost/9160
Welcome to the Cassandra CLI.
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[root@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48,
10.240.113.171]
e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200,
10.6.115.239]
[root@unknown]
$ nodetool -h localhost ring
Address DC Rack Status State Load
Owns Token
141784319550391026443072753096570088105
10.28.78.123 datacenter1 rack1 Up Normal 5.46 GB
16.67% 0
10.120.205.48 datacenter1 rack1 Up Normal 5.49 GB
16.67% 28356863910078205288614550619314017621
10.6.115.239 datacenter1 rack1 Up Normal 5.53 GB
16.67% 56713727820156410577229101238628035242
10.28.20.200 datacenter1 rack1 Up Normal 5.51 GB
16.67% 85070591730234615865843651857942052863
10.123.83.114 datacenter1 rack1 Up Normal 5.49 GB
16.67% 113427455640312821154458202477256070484
10.240.113.171 datacenter1 rack1 Up Normal 5.43 GB
16.67% 141784319550391026443072753096570088105
Daning
On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aa...@thelastpickle.com>wrote:
> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>>
> Schema may have diverged between nodes.
>
> use cassandra-cli and run describe cluster; to see how many schema
> versions you have.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>
> and we are on 0.8.6.
>
>
>
> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <da...@netseer.com> wrote:
>
>> Hi All,
>>
>>
>> We got lots of Exception in the log, and later the server crashed. any
>> idea what is happening and how to fix it?
>>
>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:4,5,main]
>> java.io.IOError: java.io.EOFException
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at
>> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>> at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>> ... 6 more
>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:2,5,main]
>> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at
>> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>> at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
>> Couldn't find cfId=-387130991
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>
>>
>>
>> The is the exception before server crashes.
>>
>>
>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[ReadRepairStage:299,5,main]
>> java.lang.RuntimeException:
>> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
>> shut down
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.RejectedExecutionException:
>> ThreadPoolExecutor has shut down
>> at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> at
>> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>> at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>> at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>> at
>> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>> at
>> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>> at
>> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>> Thank you in advance,
>>
>> Daning
>>
>
>
>
Re: Cassandra Exception
Posted by aaron morton <aa...@thelastpickle.com>.
> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
Schema may have diverged between nodes.
use cassandra-cli and run describe cluster; to see how many schema versions you have.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/03/2012, at 6:27 AM, Daning Wang wrote:
> and we are on 0.8.6.
>
>
>
> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <da...@netseer.com> wrote:
> Hi All,
>
>
> We got lots of Exception in the log, and later the server crashed. any idea what is happening and how to fix it?
>
> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[RequestResponseStage:4,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[RequestResponseStage:2,5,main]
> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=-387130991
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>
>
>
> The is the exception before server crashes.
>
>
> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadRepairStage:299,5,main]
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
> at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
> at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
> at org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
> Thank you in advance,
>
> Daning
>
Re: Cassandra Exception
Posted by Daning Wang <da...@netseer.com>.
and we are on 0.8.6.
On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <da...@netseer.com> wrote:
> Hi All,
>
>
> We got lots of Exception in the log, and later the server crashed. any
> idea what is happening and how to fix it?
>
> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[RequestResponseStage:4,5,main]
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[RequestResponseStage:2,5,main]
> java.io.IOError:
> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
> cfId=-387130991
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at
> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
> Couldn't find cfId=-387130991
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>
>
>
> The is the exception before server crashes.
>
>
> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[ReadRepairStage:299,5,main]
> java.lang.RuntimeException:
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut down
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.RejectedExecutionException:
> ThreadPoolExecutor has shut down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> at
> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
> at
> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
> at
> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
> at
> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
> Thank you in advance,
>
> Daning
>