You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sam Tunnicliffe (JIRA)" <ji...@apache.org> on 2019/07/04 09:33:01 UTC
[jira] [Comment Edited] (CASSANDRA-15199) Cassandra throwing
occasional NPE 3.11.x
[ https://issues.apache.org/jira/browse/CASSANDRA-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878488#comment-16878488 ]
Sam Tunnicliffe edited comment on CASSANDRA-15199 at 7/4/19 9:32 AM:
---------------------------------------------------------------------
Just for reference, the stacktrace from the Kong issue. Looks like the {{PagingState}} contains a null partition key, so potentially an issue with the lua client, but I seem to recall seeing some similar stacktraces when looking into paging issues in mixed version (2.1/3.0) clusters during upgrades.
{code}
WARN [ReadStage-6] 2019-05-31 01:08:07,971 ReadCommand.java:569 - Read 1000 live rows and 1003 tombstone cells for query SELECT * FROM kong_dev.routes WHERE partition = routes LIMIT 1000 (see tombstone_warn_threshold)
ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 QueryMessage.java:129 - Unexpected error during query
java.lang.NullPointerException: null
at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
{code}
was (Author: beobal):
Just for reference, the stacktrace from the Kong issue. Looks like the {PagingState} contains a null partition key, so potentially an issue with the lua client, but I seem to recall seeing some similar stacktraces when looking into paging issues in mixed version (2.1/3.0) clusters during upgrades.
{code}
WARN [ReadStage-6] 2019-05-31 01:08:07,971 ReadCommand.java:569 - Read 1000 live rows and 1003 tombstone cells for query SELECT * FROM kong_dev.routes WHERE partition = routes LIMIT 1000 (see tombstone_warn_threshold)
ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 QueryMessage.java:129 - Unexpected error during query
java.lang.NullPointerException: null
at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
{code}
> Cassandra throwing occasional NPE 3.11.x
> ----------------------------------------
>
> Key: CASSANDRA-15199
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15199
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client
> Reporter: Jeremy Justus
> Priority: Normal
>
> Hey folks, decided to raise an official Jira(never done one of these before) about an issue we have found between Kong API Gateway leveraging Cassandra as our db.
> We run a C* cluster in 2 close DCs, 6 C* nodes total, 3 in each DC. Data replicated across all nodes. We have found C* sometimes throws NPEs based on the calls made by the lua-cassandra driver Kong leverages([https://github.com/thibaultcha/lua-cassandra]). Very specifically it seems to occur when attempting to do paging across multiple C* nodes. When persistently paging to a single C* node we can't reproduce NPEs in C*.
> The exact Error C* throws with its stack-trace can be seen here:
> [https://github.com/Kong/kong/issues/4194#issuecomment-497572751]
> And again when we tried to upgrade from 3.11.2 to 3.11.4 in hopes it was already resolved:
> [https://github.com/Kong/kong/issues/4194#issuecomment-497590235]
> Same error same line numbers, so code must be same in this portion.
>
> Sample of our C* Config: [https://github.com/Kong/kong/issues/4194#issuecomment-497595766]
> We discussed this in the ASF Slack flow as well:
> ASF Slack discussion ([https://the-asf.slack.com/archives/CJZLTM05A/p1559321422028200] )
> Some of the more important technical comments I saw people posting here:
> [https://github.com/Kong/kong/issues/4194#issuecomment-497858824]
>
> I am not a C* DBA so I don't have an exact repro for you other than stating what the client application was attempting to do(Paging across multiple C* nodes within a DC) when we could see the failures. Any Apache C* folk think they see the issue or could drop me C* JAR with extra debugging print statements I could run in dev to help feed you more info? Or if you see the problem and can one shot a fix so no more NPE and Cassandra responds appropriately to the client with some sort of error message around what was wrong that would be insightful.
>
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org