You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "david.pan (JIRA)" <ji...@apache.org> on 2010/01/22 03:44:54 UTC
[jira] Created: (CASSANDRA-731) a few insert operations failed
while bootstrapping
a few insert operations failed while bootstrapping
--------------------------------------------------
Key: CASSANDRA-731
URL: https://issues.apache.org/jira/browse/CASSANDRA-731
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.5
Reporter: david.pan
Fix For: 0.6
I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
java.lang.AssertionError
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
java.lang.AssertionError
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
I traced the code and found the following assertion failed :
/* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
assert map.size() == targets.size();
The following reasons caused this issue:
1) targets is a list , not a map, as a result there may be some duplicated IP.
2) The following codes are not atomic :
org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
tokenMetadata_.updateNormalToken(token, endPoint);
calculatePendingRanges();
That's to say the IP may be both in the naturalEndpoints and pendingRanges.
eg :
targets is IPa, IPb, IPc, IPa; (size = 4)
then, the map will be IPa, IPb, IPc. (size = 3)
as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "david.pan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
david.pan updated CASSANDRA-731:
--------------------------------
Attachment: 731-throw_internal_exception_while_bootstrapping.patch
This is a simple way to deal with this problem, but it's not the final solution.
I think the best way is to make the modification of TokenMetadata atomic, but that needs a big change both in the TokenMetadata and StorageService.
Consider that the pendingRange is needed in writing, bootstrapping and leaving only, this simple modification looks like ok at current time.
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Fix For: 0.6
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-731.
--------------------------------------
Resolution: Fixed
+1 Jaakko's diagnosis. Committed assertion removal to 0.5 and trunk.
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-731:
----------------------------------------
Assignee: Jaakko Laine
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Assignee: Jaakko Laine
> Fix For: 0.6
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831379#action_12831379 ]
Jaakko Laine commented on CASSANDRA-731:
----------------------------------------
This assert might fail for the reason David said, but it might also fail because targets includes a node with pending ranges that has died (for instance did not complete bootstrap).
It does not matter if a node is in pending ranges and normal token map simultaneously, as that time is very short and may only result in a write going to too many nodes once in a while. Gossip propagation is far from being instant, so writes go to wrong places routinely anyway when nodes move. It's a much smaller thing than locking whole metadata for the duration of calculations.
I think removing the assert is enough. It should be removed in any case, since it is checking a condition that is no longer valid. I don't see any other problem in this part.
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-731:
-------------------------------------
Fix Version/s: (was: 0.6)
0.5
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805742#action_12805742 ]
Jaakko Laine commented on CASSANDRA-731:
----------------------------------------
Yeah, you're right. I'll have a look at this.
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Assignee: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-731) a few insert operations failed
while bootstrapping
Posted by "david.pan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
david.pan updated CASSANDRA-731:
--------------------------------
Attachment: the new log after patch.txt
After patch, I add a 500ms sleep between " tokenMetadata_.updateNormalToken(token, endPoint);" and " calculatePendingRanges();" to make it easy to repeat the problem.
Through the log, you can see it :-)
> a few insert operations failed while bootstrapping
> --------------------------------------------------
>
> Key: CASSANDRA-731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-731
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: david.pan
> Fix For: 0.6
>
> Attachments: 731-throw_internal_exception_while_bootstrapping.patch, the new log after patch.txt
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.
> DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
> ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
> java.lang.AssertionError
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
> at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I traced the code and found the following assertion failed :
> /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
> assert map.size() == targets.size();
> The following reasons caused this issue:
> 1) targets is a list , not a map, as a result there may be some duplicated IP.
> 2) The following codes are not atomic :
> org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
> tokenMetadata_.updateNormalToken(token, endPoint);
> calculatePendingRanges();
> That's to say the IP may be both in the naturalEndpoints and pendingRanges.
> eg :
> targets is IPa, IPb, IPc, IPa; (size = 4)
> then, the map will be IPa, IPb, IPc. (size = 3)
> as a result, assert failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.