You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yang Yang (Updated) (JIRA)" <ji...@apache.org> on 2011/10/19 21:49:10 UTC
[jira] [Updated] (CASSANDRA-3385) NPE in hinted handoff
[ https://issues.apache.org/jira/browse/CASSANDRA-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Yang updated CASSANDRA-3385:
---------------------------------
Priority: Critical (was: Major)
I just found that this contributes to another symptom I'm seeing: for RF=3, and a ring of 3 nodes, if I bring down 1 box, the remaining 2 still work fine for Quorum access, but the latency is 20x high.
I can see from debugging that a lot of time is spent on storing hints into local system table on the coordinator. but this Table.apply is slow because a lot of time is spent on the lock, while it really should not happen since the lock is sharded into 4096 ones. it turns out that all the keys used in the hints writing are the same key, at least in the examples I looked at in the debugger, if I'm correct in this observation, this is a serious bug
> NPE in hinted handoff
> ---------------------
>
> Key: CASSANDRA-3385
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3385
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Priority: Critical
>
> I'm using the current HEAD of 1.0.0 github branch, and I'm still seeing this error, not sure if it's this bug or another one.
> INFO [HintedHandoff:1] 2011-10-19 12:43:17,674 HintedHandOffManager.java (line 263) Started hinted handoff for token: 11342745564
> 0312821154458202477256070484 with IP: /10.39.85.140
> ERROR [HintedHandoff:1] 2011-10-19 12:43:17,885 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHan
> doff:1,1,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:289)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:337)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> ERROR [HintedHandoff:1] 2011-10-19 12:43:17,886 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:289)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:337)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> this could possibly be related to #3291
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira