You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Scott McPheeters <sm...@healthx.com> on 2011/05/23 20:18:10 UTC
Reboot, now node down 0.8rc1
I have a test node system running release 0.8rc1. I rebooted node3 and
now Cassandra is failing on startup.
Any ideas? I am not sure where to begin.
Debian 6, plenty of disk space, Cassandra 0.8rc1
INFO 13:48:58,192 Creating new commitlog segment
/home/cassandra/commitlog/CommitLog-1306172938192.log
INFO 13:48:58,236 Replaying
/home/cassandra/commitlog/CommitLog-1305918923361.log
INFO 13:49:04,041 Finished reading
/home/cassandra/commitlog/CommitLog-1305918923361.log
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
0)
Caused by: java.io.IOError: java.io.EOFException
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
Map.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
1)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
6)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
pListMap.java:1493)
at
java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
java:1443)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
va:402)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
FamilySerializer.java:136)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
erializer.java:126)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
wMutation.java:367)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
ndraDaemon.java:174)
at
org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
draDaemon.java:216)
... 5 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
Util.java:368)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
va:87)
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
Map.java:261)
... 17 more
Cannot load daemon
Service exit with a return value of 3
Scott
RE: Reboot, now node down 0.8rc1
Posted by Scott McPheeters <sm...@healthx.com>.
I have not changed any defaults as of yet. Yes, I do have super columns, but my gc_grace_seconds is default.
Scott
-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@datastax.com]
Sent: Tuesday, May 24, 2011 8:53 AM
To: user@cassandra.apache.org
Subject: Re: Reboot, now node down 0.8rc1
Do you have a one of your super column family with a fairly short
value for gc_grace_seconds ?
I (strongly) suspect you're hitting
https://issues.apache.org/jira//browse/CASSANDRA-2675.
--
Sylvain
On Tue, May 24, 2011 at 2:29 PM, Scott McPheeters
<sm...@healthx.com> wrote:
> It was a 0.8beta2 install that last week I upgraded to rc1.
>
> I will turn the logging up to debug the next time I have issues to get
> more details.
>
> Thank you.
>
>
>
> Scott
>
> -----Original Message-----
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, May 23, 2011 6:42 PM
> To: user@cassandra.apache.org
> Subject: Re: Reboot, now node down 0.8rc1
>
> You could have removed the affected commit log file and then run a
> nodetool repair after the node had started.
>
> It would be handy to have some more context for the problem. Was this an
> upgrade from 0.7 or a fresh install?
>
> If you are running the rc's it's handy to turn logging up to DEBUG so
> there are some more details if things go wrong.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 24 May 2011, at 08:05, Scott McPheeters wrote:
>
>> Since this is a testing system, I deleted the commit log and it came
>> right up. My question now is, let's say I had a ton of data in the
>> commit log that this node needs now. What is the best way to get the
>> data back to the node? Does a nodetool repair do this? Or do I need
> to
>> decommission the node and bring it back? Or am I missing completely
>> what the commitlog is?
>>
>>
>> Scott
>>
>>
>> -----Original Message-----
>> From: Scott McPheeters [mailto:smcpheeters@healthx.com]
>> Sent: Monday, May 23, 2011 2:18 PM
>> To: user@cassandra.apache.org
>> Subject: Reboot, now node down 0.8rc1
>>
>> I have a test node system running release 0.8rc1. I rebooted node3
> and
>> now Cassandra is failing on startup.
>>
>> Any ideas? I am not sure where to begin.
>>
>> Debian 6, plenty of disk space, Cassandra 0.8rc1
>>
>>
>> INFO 13:48:58,192 Creating new commitlog segment
>> /home/cassandra/commitlog/CommitLog-1306172938192.log
>> INFO 13:48:58,236 Replaying
>> /home/cassandra/commitlog/CommitLog-1305918923361.log
>> INFO 13:49:04,041 Finished reading
>> /home/cassandra/commitlog/CommitLog-1305918923361.log
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>> a:39)
>> at
>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>>
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
>> 0)
>> Caused by: java.io.IOError: java.io.EOFException
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
>> Map.java:265)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
>> 1)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
>> 6)
>> at
>>
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
>> pListMap.java:1493)
>> at
>>
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
>> java:1443)
>> at
>>
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
>> va:402)
>> at
>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
>> FamilySerializer.java:136)
>> at
>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
>> erializer.java:126)
>> at
>>
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
>> wMutation.java:367)
>> at
>>
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
>> at
>>
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
>> at
>>
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
>> ndraDaemon.java:174)
>> at
>>
> org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
>> draDaemon.java:216)
>> ... 5 more
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> at
>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>> at
>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
>> Util.java:368)
>> at
>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
>> va:87)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
>> Map.java:261)
>> ... 17 more
>> Cannot load daemon
>> Service exit with a return value of 3
>>
>>
>> Scott
>>
>
>
Re: Reboot, now node down 0.8rc1
Posted by Sylvain Lebresne <sy...@datastax.com>.
Do you have a one of your super column family with a fairly short
value for gc_grace_seconds ?
I (strongly) suspect you're hitting
https://issues.apache.org/jira//browse/CASSANDRA-2675.
--
Sylvain
On Tue, May 24, 2011 at 2:29 PM, Scott McPheeters
<sm...@healthx.com> wrote:
> It was a 0.8beta2 install that last week I upgraded to rc1.
>
> I will turn the logging up to debug the next time I have issues to get
> more details.
>
> Thank you.
>
>
>
> Scott
>
> -----Original Message-----
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, May 23, 2011 6:42 PM
> To: user@cassandra.apache.org
> Subject: Re: Reboot, now node down 0.8rc1
>
> You could have removed the affected commit log file and then run a
> nodetool repair after the node had started.
>
> It would be handy to have some more context for the problem. Was this an
> upgrade from 0.7 or a fresh install?
>
> If you are running the rc's it's handy to turn logging up to DEBUG so
> there are some more details if things go wrong.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 24 May 2011, at 08:05, Scott McPheeters wrote:
>
>> Since this is a testing system, I deleted the commit log and it came
>> right up. My question now is, let's say I had a ton of data in the
>> commit log that this node needs now. What is the best way to get the
>> data back to the node? Does a nodetool repair do this? Or do I need
> to
>> decommission the node and bring it back? Or am I missing completely
>> what the commitlog is?
>>
>>
>> Scott
>>
>>
>> -----Original Message-----
>> From: Scott McPheeters [mailto:smcpheeters@healthx.com]
>> Sent: Monday, May 23, 2011 2:18 PM
>> To: user@cassandra.apache.org
>> Subject: Reboot, now node down 0.8rc1
>>
>> I have a test node system running release 0.8rc1. I rebooted node3
> and
>> now Cassandra is failing on startup.
>>
>> Any ideas? I am not sure where to begin.
>>
>> Debian 6, plenty of disk space, Cassandra 0.8rc1
>>
>>
>> INFO 13:48:58,192 Creating new commitlog segment
>> /home/cassandra/commitlog/CommitLog-1306172938192.log
>> INFO 13:48:58,236 Replaying
>> /home/cassandra/commitlog/CommitLog-1305918923361.log
>> INFO 13:49:04,041 Finished reading
>> /home/cassandra/commitlog/CommitLog-1305918923361.log
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>> a:39)
>> at
>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>>
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
>> 0)
>> Caused by: java.io.IOError: java.io.EOFException
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
>> Map.java:265)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
>> 1)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
>> 6)
>> at
>>
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
>> pListMap.java:1493)
>> at
>>
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
>> java:1443)
>> at
>>
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
>> va:402)
>> at
>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
>> FamilySerializer.java:136)
>> at
>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
>> erializer.java:126)
>> at
>>
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
>> wMutation.java:367)
>> at
>>
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
>> at
>>
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
>> at
>>
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
>> ndraDaemon.java:174)
>> at
>>
> org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
>> draDaemon.java:216)
>> ... 5 more
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> at
>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>> at
>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
>> Util.java:368)
>> at
>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
>> va:87)
>> at
>>
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
>> Map.java:261)
>> ... 17 more
>> Cannot load daemon
>> Service exit with a return value of 3
>>
>>
>> Scott
>>
>
>
RE: Reboot, now node down 0.8rc1
Posted by Scott McPheeters <sm...@healthx.com>.
It was a 0.8beta2 install that last week I upgraded to rc1.
I will turn the logging up to debug the next time I have issues to get
more details.
Thank you.
Scott
-----Original Message-----
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, May 23, 2011 6:42 PM
To: user@cassandra.apache.org
Subject: Re: Reboot, now node down 0.8rc1
You could have removed the affected commit log file and then run a
nodetool repair after the node had started.
It would be handy to have some more context for the problem. Was this an
upgrade from 0.7 or a fresh install?
If you are running the rc's it's handy to turn logging up to DEBUG so
there are some more details if things go wrong.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 24 May 2011, at 08:05, Scott McPheeters wrote:
> Since this is a testing system, I deleted the commit log and it came
> right up. My question now is, let's say I had a ton of data in the
> commit log that this node needs now. What is the best way to get the
> data back to the node? Does a nodetool repair do this? Or do I need
to
> decommission the node and bring it back? Or am I missing completely
> what the commitlog is?
>
>
> Scott
>
>
> -----Original Message-----
> From: Scott McPheeters [mailto:smcpheeters@healthx.com]
> Sent: Monday, May 23, 2011 2:18 PM
> To: user@cassandra.apache.org
> Subject: Reboot, now node down 0.8rc1
>
> I have a test node system running release 0.8rc1. I rebooted node3
and
> now Cassandra is failing on startup.
>
> Any ideas? I am not sure where to begin.
>
> Debian 6, plenty of disk space, Cassandra 0.8rc1
>
>
> INFO 13:48:58,192 Creating new commitlog segment
> /home/cassandra/commitlog/CommitLog-1306172938192.log
> INFO 13:48:58,236 Replaying
> /home/cassandra/commitlog/CommitLog-1305918923361.log
> INFO 13:49:04,041 Finished reading
> /home/cassandra/commitlog/CommitLog-1305918923361.log
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
> 0)
> Caused by: java.io.IOError: java.io.EOFException
> at
>
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
> Map.java:265)
> at
>
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
> 1)
> at
>
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
> 6)
> at
>
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
> pListMap.java:1493)
> at
>
java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
> java:1443)
> at
>
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
> va:402)
> at
>
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
> FamilySerializer.java:136)
> at
>
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
> erializer.java:126)
> at
>
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
> wMutation.java:367)
> at
>
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
> at
>
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at
>
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
> ndraDaemon.java:174)
> at
>
org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
> draDaemon.java:216)
> ... 5 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
>
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
>
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
> Util.java:368)
> at
>
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
> va:87)
> at
>
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
> Map.java:261)
> ... 17 more
> Cannot load daemon
> Service exit with a return value of 3
>
>
> Scott
>
Re: Reboot, now node down 0.8rc1
Posted by aaron morton <aa...@thelastpickle.com>.
You could have removed the affected commit log file and then run a nodetool repair after the node had started.
It would be handy to have some more context for the problem. Was this an upgrade from 0.7 or a fresh install?
If you are running the rc's it's handy to turn logging up to DEBUG so there are some more details if things go wrong.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 24 May 2011, at 08:05, Scott McPheeters wrote:
> Since this is a testing system, I deleted the commit log and it came
> right up. My question now is, let's say I had a ton of data in the
> commit log that this node needs now. What is the best way to get the
> data back to the node? Does a nodetool repair do this? Or do I need to
> decommission the node and bring it back? Or am I missing completely
> what the commitlog is?
>
>
> Scott
>
>
> -----Original Message-----
> From: Scott McPheeters [mailto:smcpheeters@healthx.com]
> Sent: Monday, May 23, 2011 2:18 PM
> To: user@cassandra.apache.org
> Subject: Reboot, now node down 0.8rc1
>
> I have a test node system running release 0.8rc1. I rebooted node3 and
> now Cassandra is failing on startup.
>
> Any ideas? I am not sure where to begin.
>
> Debian 6, plenty of disk space, Cassandra 0.8rc1
>
>
> INFO 13:48:58,192 Creating new commitlog segment
> /home/cassandra/commitlog/CommitLog-1306172938192.log
> INFO 13:48:58,236 Replaying
> /home/cassandra/commitlog/CommitLog-1305918923361.log
> INFO 13:49:04,041 Finished reading
> /home/cassandra/commitlog/CommitLog-1305918923361.log
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
> 0)
> Caused by: java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
> Map.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
> 1)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
> 6)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
> pListMap.java:1493)
> at
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
> java:1443)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
> va:402)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
> FamilySerializer.java:136)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
> erializer.java:126)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
> wMutation.java:367)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
> ndraDaemon.java:174)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
> draDaemon.java:216)
> ... 5 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
> Util.java:368)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
> va:87)
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
> Map.java:261)
> ... 17 more
> Cannot load daemon
> Service exit with a return value of 3
>
>
> Scott
>
RE: Reboot, now node down 0.8rc1
Posted by Scott McPheeters <sm...@healthx.com>.
Since this is a testing system, I deleted the commit log and it came
right up. My question now is, let's say I had a ton of data in the
commit log that this node needs now. What is the best way to get the
data back to the node? Does a nodetool repair do this? Or do I need to
decommission the node and bring it back? Or am I missing completely
what the commitlog is?
Scott
-----Original Message-----
From: Scott McPheeters [mailto:smcpheeters@healthx.com]
Sent: Monday, May 23, 2011 2:18 PM
To: user@cassandra.apache.org
Subject: Reboot, now node down 0.8rc1
I have a test node system running release 0.8rc1. I rebooted node3 and
now Cassandra is failing on startup.
Any ideas? I am not sure where to begin.
Debian 6, plenty of disk space, Cassandra 0.8rc1
INFO 13:48:58,192 Creating new commitlog segment
/home/cassandra/commitlog/CommitLog-1306172938192.log
INFO 13:48:58,236 Replaying
/home/cassandra/commitlog/CommitLog-1305918923361.log
INFO 13:49:04,041 Finished reading
/home/cassandra/commitlog/CommitLog-1305918923361.log
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:16
0)
Caused by: java.io.IOError: java.io.EOFException
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
Map.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:28
1)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:23
6)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSki
pListMap.java:1493)
at
java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.
java:1443)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.ja
va:402)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Column
FamilySerializer.java:136)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilyS
erializer.java:126)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Ro
wMutation.java:367)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:271)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassa
ndraDaemon.java:174)
at
org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassan
draDaemon.java:216)
... 5 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBuffer
Util.java:368)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.ja
va:87)
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSorted
Map.java:261)
... 17 more
Cannot load daemon
Service exit with a return value of 3
Scott