You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dan Di Spaltro (JIRA)" <ji...@apache.org> on 2010/01/20 05:39:56 UTC
[jira] Created: (CASSANDRA-722) batch insert failing
batch insert failing
--------------------
Key: CASSANDRA-722
URL: https://issues.apache.org/jira/browse/CASSANDRA-722
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.5
Environment: r900058 4 node cluster, 1 more bootstrapping
Reporter: Dan Di Spaltro
I get this during the course of an insert.
ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-722:
-----------------------------------
Attachment: 722.patch
yeah, proximity seems good.
Attached patch ignores dead bootstrappers and looks for hinted locations based on proximity (list of addresses is from gossiper.liveMembers).
This couples ARS with Gossiper, but don't know if that matters so much as it used FD already before.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802844#action_12802844 ]
Jaakko Laine commented on CASSANDRA-722:
----------------------------------------
Is it really OK to send hints to random targets? That will scatter hinted data around the cluster as each hinted write may go to different location. Might not be problematic, but feels like we're going around the problem instead of fixing the cause. Something would certainly be gained by having the load spread amongst nodes, but isn't something also lost if (potentially) all nodes in the cluster stream hinted data instead of one node?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Dan Di Spaltro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803008#action_12803008 ]
Dan Di Spaltro commented on CASSANDRA-722:
------------------------------------------
Should I add another bug for the failed bootstrapping problem?
The node seems to be getting data; like it thinks its part of the cluster and datagrowth is that of a normal working node. However, it doesn't have all the data (should be at least 50g and it 3 gb) and if I try to nodeprobe ring it doesn't show up.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-722) batch insert failing
with TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802860#action_12802860 ]
Jaakko Laine edited comment on CASSANDRA-722 at 1/20/10 3:13 PM:
-----------------------------------------------------------------
yeah, proximity seems good.
Attached patch ignores dead bootstrappers and looks for hinted locations based on proximity (list of addresses is from gossiper.liveMembers).
This couples ARS with Gossiper, but don't know if that matters so much as it used FD already before.
Edit: this patch has not been tested much, have to do some more testing tomorrow.
was (Author: jaakko):
yeah, proximity seems good.
Attached patch ignores dead bootstrappers and looks for hinted locations based on proximity (list of addresses is from gossiper.liveMembers).
This couples ARS with Gossiper, but don't know if that matters so much as it used FD already before.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802658#action_12802658 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
have you done any node movement in this cluster?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803113#action_12803113 ]
Jaakko Laine commented on CASSANDRA-722:
----------------------------------------
> Is there a way to remove the failed bootstrapped node from the cluster, decommission obviously doesn't work...
Actually there is an "undocumented" feature in removetoken command that it will clear the token from bootstrapping as well. If you know the token, remove it and that should take care of it.
Probably need to provide better tools to investigate/poke bootstrapping and leaving tokens.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802694#action_12802694 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
We could fix that by picking a random node to start our scan of potential hint destinations; there's nothing magical about starting w/ endpoint+1. In fact picking random is better, since currently having a node down degrades performance more than it should since all the hints go to the same node.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802680#action_12802680 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
(1) definitely
(2) Under what conditions does it make sense to ask for Token of a node that is not a ring member?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802999#action_12802999 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
this will fix your batch_insert problem
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-722:
-----------------------------------
Attachment: (was: 722.patch)
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Dan Di Spaltro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802987#action_12802987 ]
Dan Di Spaltro commented on CASSANDRA-722:
------------------------------------------
Is there a way to remove the failed bootstrapped node from the cluster, decommission obviously doesn't work...
Or should this patch fix things?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-722:
-----------------------------------
Attachment: (was: 722.patch)
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-722:
-------------------------------------
Component/s: Core
Fix Version/s: 0.5
Assignee: Jaakko Laine
Summary: batch insert failing with TokenMetadata AssertionError (was: batch insert failing)
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803125#action_12803125 ]
Jaakko Laine commented on CASSANDRA-722:
----------------------------------------
yeah, 644 is needed to remove it from gossip, but it will be removed from token metadata even without 644, which is what matters in this case.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802675#action_12802675 ]
Jaakko Laine commented on CASSANDRA-722:
----------------------------------------
Seems there's a bug in getHintedMapForEndpoints.
That bootsrapping node is probably considered dead by other nodes
This is what's happening:
(1) Node bootstraps -> it has pending ranges, but it is not member of token ring
(2) During write, pending ranges match -> node is added to write endpoints
(3) Node is down -> a hinted target will be searched for
(4) getToken explodes because endpoint is not member
There are two things that need to be considered:
(1) We probably should ignore endpoints that are not members when looking for hinted targets
(2) That assert in getToken is a bit outdated I think. Nodes may come and go during the time ARS and friends are looking for write targets. It might very well happen that a node was removed between getting natural endpoints and coming back to get hinted targets. There's a comment saying "don't want to return nulls", but perhaps we'll need to reconsider this?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-722:
-----------------------------------
Attachment: 722.patch
new version. skip non-members also when considering hinted location.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine resolved CASSANDRA-722.
------------------------------------
Resolution: Fixed
committed to 0.5.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Dan Di Spaltro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802667#action_12802667 ]
Dan Di Spaltro commented on CASSANDRA-722:
------------------------------------------
I have used loadbalance quite a few times. I have never done a decommision or a move command explicitly.
A machine does seem to be stalled in the middle of bootstrapping. Trying to get intellij setup, to dig deeper on to what that machine is doing.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802863#action_12802863 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
+1
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-722:
-----------------------------------
Attachment: 722.patch
accidentally submitted a japanese version, this should be better.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802848#action_12802848 ]
Jonathan Ellis commented on CASSANDRA-722:
------------------------------------------
no, we don't rely on hint location.
but actually using sortByProximity is probably best of all.
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Dan Di Spaltro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803119#action_12803119 ]
Dan Di Spaltro commented on CASSANDRA-722:
------------------------------------------
I think jonathan mentioned that I would need the patch rom 644 is that correct?
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 722.patch
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-722) batch insert failing with
TokenMetadata AssertionError
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802690#action_12802690 ]
Jaakko Laine commented on CASSANDRA-722:
----------------------------------------
(2) it does not make sense, but it might happen. Suppose a node is natural endpoint of a write. StorageProxy gets these in mutate, and then passes the same list to getHintedEndpointMap. If one of these natural endpoints is removed in between getNaturalEndpoints and finding hinted targets, there will be a call to getToken for non-existing endpoint. It does not help even if we insert isMember check right before getToken, as there is still small window that could cause this.
Basically the same can happen wherever we use getToken
> batch insert failing with TokenMetadata AssertionError
> ------------------------------------------------------
>
> Key: CASSANDRA-722
> URL: https://issues.apache.org/jira/browse/CASSANDRA-722
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: r900058 4 node cluster, 1 more bootstrapping
> Reporter: Dan Di Spaltro
> Assignee: Jaakko Laine
> Fix For: 0.5
>
>
> I get this during the course of an insert.
> ERROR [pool-1-thread-17] 2010-01-20 03:50:40,517 Cassandra.java (line 1096) Internal error processing batch_insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
> at org.apache.cassandra.service.StorageProxy.insert(StorageProxy.java:101)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:470)
> at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
> at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.