You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mohica Jasha <mo...@gmail.com> on 2014/05/05 23:01:10 UTC
error in cassandra log file under stress
one query in our prod system under heavy load failed.
I cant tell which query it was. But I don't think we were using an invalid
consistency level since our configuration work all the time except when it
goes under heavy load.
If I can reproduce this I will pass more information.
I wonder if this is due to a race condition in Paxos implementation.
I found the following in the cassandra's log file:
ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>> QueryMessage.java (line 131) Unexpected error during query
>
> java.lang.UnsupportedOperationException: Invalid consistency level:
>> LOCAL_SERIAL
>
> at
>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>
> at
>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>
> at
>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>
> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>
> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>
> at
>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>
> at
>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>
> at
>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>
> at
>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>
> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>
> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>
> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>
> at
>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>
> at
>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>
> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>> ErrorMessage.java (line 222) Unexpected exception during request
>
> java.lang.UnsupportedOperationException: Invalid consistency level:
>> LOCAL_SERIAL
>
> at
>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>
> at
>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>
> at
>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>
> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>
> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>
> at
>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>
> at
>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>
> at
>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>
> at
>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>
> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>
> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>
> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>
> at
>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>
> at
>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>
> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
>
Re: error in cassandra log file under stress
Posted by Mohica Jasha <mo...@gmail.com>.
Most likely it is the following query:
DELETE from conditional_update_lock where resource_id = '${myresourceid}' IF
lock_id = ${myuuid};
We are using datastax 2.0.1 and the datastax threw a ReadtimeoutException
On Mon, May 5, 2014 at 5:03 PM, Mohica Jasha <mo...@gmail.com> wrote:
> forgot to say which Cassandra version we are using.
>
> Our prod deployment:
>
> C* 2.0.5, DC1:3, DC2:3
>
>
>
>
> On Mon, May 5, 2014 at 5:01 PM, Mohica Jasha <mo...@gmail.com>wrote:
>
>> one query in our prod system under heavy load failed.
>> I cant tell which query it was. But I don't think we were using an
>> invalid consistency level since our configuration work all the time except
>> when it goes under heavy load.
>> If I can reproduce this I will pass more information.
>>
>> I wonder if this is due to a race condition in Paxos implementation.
>> I found the following in the cassandra's log file:
>>
>>
>> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>>> QueryMessage.java (line 131) Unexpected error during query
>>>
>>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>>> LOCAL_SERIAL
>>>
>>> at
>>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>>
>>> at
>>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>>
>>> at
>>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>>
>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>>
>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>>
>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>>
>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>>
>>> at
>>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>>
>>> at
>>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>>
>>> at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>>
>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>
>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>>
>>> at
>>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>>
>>> at
>>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>>> ErrorMessage.java (line 222) Unexpected exception during request
>>>
>>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>>> LOCAL_SERIAL
>>>
>>> at
>>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>>
>>> at
>>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>>
>>> at
>>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>>
>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>>
>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>>
>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>>
>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>>
>>> at
>>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>>
>>> at
>>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>>
>>> at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>>
>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>
>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>>
>>> at
>>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>>
>>> at
>>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>
Re: error in cassandra log file under stress
Posted by Mohica Jasha <mo...@gmail.com>.
Thanks Robert,
I understand that C* < 2.1 is probably not perfect for prod. But the choice
to use C* 2.0.x was made long ago and we are trying to keep up.
Anyway I filled a jira ticket:
https://issues.apache.org/jira/browse/CASSANDRA-7176
On Mon, May 5, 2014 at 5:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, May 5, 2014 at 2:03 PM, Mohica Jasha <mo...@gmail.com>wrote:
>
>> Our prod deployment:
>>
>> C* 2.0.5, DC1:3, DC2:3
>>
>
> Not directly related to your bug report, which I would file an Apache JIRA
> regarding, but...
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
> =Rob
>
>
Re: error in cassandra log file under stress
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, May 5, 2014 at 2:03 PM, Mohica Jasha <mo...@gmail.com> wrote:
> Our prod deployment:
>
> C* 2.0.5, DC1:3, DC2:3
>
Not directly related to your bug report, which I would file an Apache JIRA
regarding, but...
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
=Rob
Re: error in cassandra log file under stress
Posted by Mohica Jasha <mo...@gmail.com>.
forgot to say which Cassandra version we are using.
Our prod deployment:
C* 2.0.5, DC1:3, DC2:3
On Mon, May 5, 2014 at 5:01 PM, Mohica Jasha <mo...@gmail.com> wrote:
> one query in our prod system under heavy load failed.
> I cant tell which query it was. But I don't think we were using an invalid
> consistency level since our configuration work all the time except when it
> goes under heavy load.
> If I can reproduce this I will pass more information.
>
> I wonder if this is due to a race condition in Paxos implementation.
> I found the following in the cassandra's log file:
>
>
> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>> QueryMessage.java (line 131) Unexpected error during query
>>
>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>> LOCAL_SERIAL
>>
>> at
>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>
>> at
>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>
>> at
>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>
>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>
>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>
>> at
>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>
>> at
>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>
>> at
>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>
>> at
>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>
>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>
>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>
>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>
>> at
>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>
>> at
>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>
>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>> ErrorMessage.java (line 222) Unexpected exception during request
>>
>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>> LOCAL_SERIAL
>>
>> at
>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>
>> at
>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>
>> at
>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>
>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>
>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>
>> at
>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>
>> at
>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>
>> at
>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>
>> at
>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>
>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>
>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>
>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>
>> at
>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>
>> at
>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>
>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>>