You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stu King <st...@stuartrexking.com> on 2011/02/04 09:18:10 UTC
Move
I am running a move on one node in a 5 node cluster. There are no writes to
the cluster during the move.
I am seeing an exception on one of the nodes (not the node which I am doing
the move on).
The exception stack is
ERROR [CompactionExecutor:1] 2011-02-04 08:10:46,855 PrecompactedRow.java
(line 82) Skipping row DecoratedKey(656517988577125179070965247963445,
555345524e414d452e6a6f746173696c766573747265) in
/var/lib/cassandra/data/Wenzani/UUID_UUID_SUPER-e-408-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Output from nodetool ring.
Address Status State Load Owns Token
105916716988735575505223832861775432335
1.1.1.2 Up Normal 34.29 GB 45.36%
12956529933298582072612274413196299151
1.1.1.3 Up Normal 34.46 GB 11.41%
32366675628954067180152712803029297247
1.1.1.4 Up Normal 48.96 GB 11.40%
51756081624280481651195537730585467204
1.1.1.5 Up Normal 22 GB 22.78%
90515859237527157456212262236145255573
1.1.1.6 Up Leaving 13.34 GB 9.05%
105916716988735575505223832861775432335
1.1.1.6 is the node which I executed the move on. It seems to be locked in
the Leaving state. Is this normal until the move completes?
There is almost no activity in the logs and very little cpu usage across the
cluster.
Is this expected for a move?
Cheers
Stu
Re: Move
Posted by Jonathan Ellis <jb...@gmail.com>.
Looks like https://issues.apache.org/jira/browse/CASSANDRA-1992, fixed
for 0.7.1.
On Fri, Feb 4, 2011 at 12:18 AM, Stu King <st...@stuartrexking.com> wrote:
> I am running a move on one node in a 5 node cluster. There are no writes to
> the cluster during the move.
> I am seeing an exception on one of the nodes (not the node which I am doing
> the move on).
> The exception stack is
> ERROR [CompactionExecutor:1] 2011-02-04 08:10:46,855 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(656517988577125179070965247963445,
> 555345524e414d452e6a6f746173696c766573747265) in
> /var/lib/cassandra/data/Wenzani/UUID_UUID_SUPER-e-408-Data.db
> java.io.EOFException
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
> at
> org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
> at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Output from nodetool ring.
> Address Status State Load Owns Token
>
>
> 105916716988735575505223832861775432335
> 1.1.1.2 Up Normal 34.29 GB 45.36%
> 12956529933298582072612274413196299151
> 1.1.1.3 Up Normal 34.46 GB 11.41%
> 32366675628954067180152712803029297247
> 1.1.1.4 Up Normal 48.96 GB 11.40%
> 51756081624280481651195537730585467204
> 1.1.1.5 Up Normal 22 GB 22.78%
> 90515859237527157456212262236145255573
> 1.1.1.6 Up Leaving 13.34 GB 9.05%
> 105916716988735575505223832861775432335
> 1.1.1.6 is the node which I executed the move on. It seems to be locked in
> the Leaving state. Is this normal until the move completes?
> There is almost no activity in the logs and very little cpu usage across the
> cluster.
> Is this expected for a move?
> Cheers
> Stu
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com