You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "B. Todd Burruss" <bb...@real.com> on 2010/08/23 20:33:51 UTC
0.7 beta 1 - "error in row mutation" and NPE
i see the following in my server logs quite closely while doing a lot of
batch_mutations and reads. i create keyspaces and column families using
thrift api, not cassandra.yaml. did not migrate anything from 0.6.
4 node cluster, RF = 3, QUORUM read/write.
happens immediately on a fresh start of the cluster with no data -
removed all data and commit logs before starting.
i believe the problem is because the creation of the keyspace/CFs has
possible not propagated to all nodes before i start slamming with the
batch_mutates and gets. if i put a 1 second sleep after the keyspace
has been created, but before i start using it, all is well.
ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659
RowMutationVerbHandler.java (line 78) Error in row mutation
org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't
find cfId=1001
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115)
at
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197
DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
java.lang.NullPointerException
at org.apache.cassandra.db.Table.open(Table.java:104)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Re: 0.7 beta 1 - "error in row mutation" and NPE
Posted by "B. Todd Burruss" <bb...@real.com>.
yes, if i wait a second or two all is well. i didn't know about
"check_schema_agreement". is there a wiki page about using the new
keyspace management API? i don't see it on the Thrift API page.
i will apply the other patch you mentioned as well.
i think this is the same scenario,
https://issues.apache.org/jira/browse/CASSANDRA-1335
thx
On Mon, 2010-08-23 at 12:08 -0700, Gary Dusbabek wrote:
> On second thought, that doesn't look like the problem at all.
>
> Have you waited long enough for the schema to propagate? What does
> check_schema_agreement tell you?
>
> One thing to make sure about: you should funnel all your schema
> updates through a single coordinator node or make sure that the
> cluster is quiescent before sending schema updates to a random node.
>
> Gary.
>
>
> On Mon, Aug 23, 2010 at 14:03, Gary Dusbabek <gd...@gmail.com> wrote:
> > It looks like you're running into
> > https://issues.apache.org/jira/browse/CASSANDRA-1403, which was fixed
> > last week and will be included in beta2.
> >
> > If you are experiencing this on trunk, please do file another ticket,
> > or comment on the existing one.
> >
> > Gary.
> >
> >
> > On Mon, Aug 23, 2010 at 13:33, B. Todd Burruss <bb...@real.com> wrote:
> >> i see the following in my server logs quite closely while doing a lot of
> >> batch_mutations and reads. i create keyspaces and column families using
> >> thrift api, not cassandra.yaml. did not migrate anything from 0.6.
> >>
> >> 4 node cluster, RF = 3, QUORUM read/write.
> >>
> >> happens immediately on a fresh start of the cluster with no data - removed
> >> all data and commit logs before starting.
> >>
> >> i believe the problem is because the creation of the keyspace/CFs has
> >> possible not propagated to all nodes before i start slamming with the
> >> batch_mutates and gets. if i put a 1 second sleep after the keyspace has
> >> been created, but before i start using it, all is well.
> >>
> >>
> >> ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659
> >> RowMutationVerbHandler.java (line 78) Error in row mutation
> >> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
> >> cfId=1001
> >> at
> >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115)
> >> at
> >> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372)
> >> at
> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382)
> >> at
> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340)
> >> at
> >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
> >> at
> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >> ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197
> >> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
> >> java.lang.NullPointerException
> >> at org.apache.cassandra.db.Table.open(Table.java:104)
> >> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
> >> at
> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >>
> >
Re: 0.7 beta 1 - "error in row mutation" and NPE
Posted by Gary Dusbabek <gd...@gmail.com>.
On second thought, that doesn't look like the problem at all.
Have you waited long enough for the schema to propagate? What does
check_schema_agreement tell you?
One thing to make sure about: you should funnel all your schema
updates through a single coordinator node or make sure that the
cluster is quiescent before sending schema updates to a random node.
Gary.
On Mon, Aug 23, 2010 at 14:03, Gary Dusbabek <gd...@gmail.com> wrote:
> It looks like you're running into
> https://issues.apache.org/jira/browse/CASSANDRA-1403, which was fixed
> last week and will be included in beta2.
>
> If you are experiencing this on trunk, please do file another ticket,
> or comment on the existing one.
>
> Gary.
>
>
> On Mon, Aug 23, 2010 at 13:33, B. Todd Burruss <bb...@real.com> wrote:
>> i see the following in my server logs quite closely while doing a lot of
>> batch_mutations and reads. i create keyspaces and column families using
>> thrift api, not cassandra.yaml. did not migrate anything from 0.6.
>>
>> 4 node cluster, RF = 3, QUORUM read/write.
>>
>> happens immediately on a fresh start of the cluster with no data - removed
>> all data and commit logs before starting.
>>
>> i believe the problem is because the creation of the keyspace/CFs has
>> possible not propagated to all nodes before i start slamming with the
>> batch_mutates and gets. if i put a 1 second sleep after the keyspace has
>> been created, but before i start using it, all is well.
>>
>>
>> ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659
>> RowMutationVerbHandler.java (line 78) Error in row mutation
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=1001
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115)
>> at
>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372)
>> at
>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382)
>> at
>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340)
>> at
>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197
>> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
>> java.lang.NullPointerException
>> at org.apache.cassandra.db.Table.open(Table.java:104)
>> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>
Re: 0.7 beta 1 - "error in row mutation" and NPE
Posted by Gary Dusbabek <gd...@gmail.com>.
It looks like you're running into
https://issues.apache.org/jira/browse/CASSANDRA-1403, which was fixed
last week and will be included in beta2.
If you are experiencing this on trunk, please do file another ticket,
or comment on the existing one.
Gary.
On Mon, Aug 23, 2010 at 13:33, B. Todd Burruss <bb...@real.com> wrote:
> i see the following in my server logs quite closely while doing a lot of
> batch_mutations and reads. i create keyspaces and column families using
> thrift api, not cassandra.yaml. did not migrate anything from 0.6.
>
> 4 node cluster, RF = 3, QUORUM read/write.
>
> happens immediately on a fresh start of the cluster with no data - removed
> all data and commit logs before starting.
>
> i believe the problem is because the creation of the keyspace/CFs has
> possible not propagated to all nodes before i start slamming with the
> batch_mutates and gets. if i put a 1 second sleep after the keyspace has
> been created, but before i start using it, all is well.
>
>
> ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659
> RowMutationVerbHandler.java (line 78) Error in row mutation
> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
> cfId=1001
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115)
> at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340)
> at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197
> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
> java.lang.NullPointerException
> at org.apache.cassandra.db.Table.open(Table.java:104)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
>