You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/05/05 09:00:30 UTC

[jira] Commented: (CASSANDRA-133) Local web interface hangs

    [ https://issues.apache.org/jira/browse/CASSANDRA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705928#action_12705928 ] 

Jonathan Ellis commented on CASSANDRA-133:
------------------------------------------

I don't see anything obvious in the thread dump -- it looks like the request completed!

I added some more logging in r771591.  Give that a try.  Here is what it should look like:

DEBUG - Processing http request from socket ...                     
DEBUG - Startline method=GET path=/ query= version=HTTP/1.1         
DEBUG - Host : localhost:7002                                       
DEBUG - User-Agent : Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/8.10 (intrepid) Firefox/3.0.10                                                      
DEBUG - Accept : text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8           
DEBUG - Processing http request from socket ...                                            
DEBUG - Accept-Language : en-us,en;q=0.5                                                   
DEBUG - Accept-Encoding : gzip,deflate                                                     
DEBUG - Accept-Charset : ISO-8859-1,utf-8;q=0.7,*;q=0.7                                    
DEBUG - Keep-Alive : 300                                                                   
DEBUG - Connection : keep-alive                                                            
DEBUG - ... done parsing request for /                                                     
DEBUG - Handling GET                                                                       
DEBUG - Constructing range to endpoint map ...                                             
DEBUG - Done constructing range to endpoint map ...                                        
DEBUG - http response is 6651 bytes                                                        
INFO - Closing HTTP socket ...                                                             

> Local web interface hangs
> -------------------------
>
>                 Key: CASSANDRA-133
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-133
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: trunk
>         Environment: Mac OS X 10.5
>            Reporter: Evan Weaver
>
> DEBUG - Processing http requests from socket ...
> DEBUG - Startline method=GET path=/ query= version=HTTP/1.1
> DEBUG - User-Agent : Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
> DEBUG - Accept : application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
> 2009-05-04 17:09:55
> Full thread dump Java HotSpot(TM) Server VM (1.6.0_03-p3-landonf_03_feb_2008_02_12-b00 mixed mode):
> "HTTP-CONNECTION:1" prio=5 tid=0x00936800 nid=0xb1aef000 waiting on condition [0xb1aee000..0xb1aee8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04fd67c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "GMFD:1" prio=5 tid=0x00935400 nid=0xb1a9d000 waiting on condition [0xb1a9c000..0xb1a9c958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ff6098> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "Timer-1" prio=5 tid=0x0092f000 nid=0xb1a4b000 in Object.wait() [0xb1a4a000..0xb1a4a9d8]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x04fbab38> (a java.util.TaskQueue)
> 	at java.util.TimerThread.mainLoop(Timer.java:509)
> 	- locked <0x04fbab38> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring AnalyticsContext" daemon prio=5 tid=0x0092ec00 nid=0xb19f9000 in Object.wait() [0xb19f8000..0xb19f8a58]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x04fa9350> (a java.util.TaskQueue)
> 	at java.util.TimerThread.mainLoop(Timer.java:509)
> 	- locked <0x04fa9350> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:462)
> "UDP Selector Manager" prio=5 tid=0x00935000 nid=0xb19a7000 runnable [0xb19a6000..0xb19a6ad8]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
> 	at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
> 	at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 	- locked <0x04f98c00> (a sun.nio.ch.Util$1)
> 	- locked <0x04f98bf0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x04f989f8> (a sun.nio.ch.PollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> 	at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:92)
> "TCP Selector Manager" prio=5 tid=0x00934800 nid=0xb1955000 runnable [0xb1954000..0xb1954b58]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
> 	at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
> 	at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> 	- locked <0x04f8d018> (a sun.nio.ch.Util$1)
> 	- locked <0x04f8d008> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x04f8ce00> (a sun.nio.ch.PollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> 	at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:92)
> "HINTED-HANDOFF-POOL:1" prio=5 tid=0x00931000 nid=0xb1903000 waiting on condition [0xb1902000..0xb1902bd8]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04eebb78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> 	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MINOR-COMPACTION-POOL:1" prio=5 tid=0x0092cc00 nid=0xb18b1000 waiting on condition [0xb18b0000..0xb18b0858]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04e1e330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> 	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x0092a000 nid=0xb185f000 waiting on condition [0xb185e000..0xb185e8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04df4838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00928400 nid=0xb180d000 waiting on condition [0xb180c000..0xb180c958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04df3088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00929400 nid=0xb17bb000 waiting on condition [0xb17ba000..0xb17ba9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04df18d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00928c00 nid=0xb1769000 waiting on condition [0xb1768000..0xb1768a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04df0058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00927800 nid=0xb1717000 waiting on condition [0xb1716000..0xb1716ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04dee8c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00923400 nid=0xb16c5000 waiting on condition [0xb16c4000..0xb16c4b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ded138> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00925800 nid=0xb1673000 waiting on condition [0xb1672000..0xb1672bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04deb9c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00923c00 nid=0xb1621000 waiting on condition [0xb1620000..0xb1620858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04dea238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00921c00 nid=0xb15cf000 waiting on condition [0xb15ce000..0xb15ce8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04de8ac8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "FAST-MEMTABLE-POOL:1" prio=5 tid=0x00922400 nid=0xb157d000 waiting on condition [0xb157c000..0xb157c958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04de1e10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "HTTP-REQUEST:1" prio=5 tid=0x0091d000 nid=0xb152b000 waiting on condition [0xb152a000..0xb152a9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d3b5f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MAP-REDUCE-STAGE:4" prio=5 tid=0x0091c000 nid=0xb14d9000 waiting on condition [0xb14d8000..0xb14d8a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d3a990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MAP-REDUCE-STAGE:3" prio=5 tid=0x0091b400 nid=0xb1487000 waiting on condition [0xb1486000..0xb1486ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d3a990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MAP-REDUCE-STAGE:2" prio=5 tid=0x0091a400 nid=0xb1435000 waiting on condition [0xb1434000..0xb1434b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d3a990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MAP-REDUCE-STAGE:1" prio=5 tid=0x00919800 nid=0xb13e3000 waiting on condition [0xb13e2000..0xb13e2bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d3a990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:8" prio=5 tid=0x00918800 nid=0xb1391000 waiting on condition [0xb1390000..0xb1390858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:7" prio=5 tid=0x00917c00 nid=0xb133f000 waiting on condition [0xb133e000..0xb133e8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:6" prio=5 tid=0x00916c00 nid=0xb12ed000 waiting on condition [0xb12ec000..0xb12ec958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:5" prio=5 tid=0x00916000 nid=0xb129b000 waiting on condition [0xb129a000..0xb129a9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:4" prio=5 tid=0x00915000 nid=0xb1249000 waiting on condition [0xb1248000..0xb1248a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:3" prio=5 tid=0x00914400 nid=0xb11f7000 waiting on condition [0xb11f6000..0xb11f6ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:2" prio=5 tid=0x00913400 nid=0xb11a5000 waiting on condition [0xb11a4000..0xb11a4b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-READ-STAGE:1" prio=5 tid=0x00912800 nid=0xb1153000 waiting on condition [0xb1152000..0xb1152bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-MUTATION-STAGE:4" prio=5 tid=0x00911800 nid=0xb1101000 waiting on condition [0xb1100000..0xb1100858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d387d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-MUTATION-STAGE:3" prio=5 tid=0x00910c00 nid=0xb10af000 waiting on condition [0xb10ae000..0xb10ae8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d387d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-MUTATION-STAGE:2" prio=5 tid=0x0090fc00 nid=0xb105d000 waiting on condition [0xb105c000..0xb105c958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d387d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "ROW-MUTATION-STAGE:1" prio=5 tid=0x0090f000 nid=0xb100b000 waiting on condition [0xb100a000..0xb100a9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d387d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "CONSISTENCY-MANAGER:4" prio=5 tid=0x0090e000 nid=0xb0fb9000 waiting on condition [0xb0fb8000..0xb0fb8a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d37b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "CONSISTENCY-MANAGER:3" prio=5 tid=0x0090d800 nid=0xb0f67000 waiting on condition [0xb0f66000..0xb0f66ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d37b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "CONSISTENCY-MANAGER:2" prio=5 tid=0x0090d000 nid=0xb0f15000 waiting on condition [0xb0f14000..0xb0f14b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d37b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "CONSISTENCY-MANAGER:1" prio=5 tid=0x0090cc00 nid=0xb0ec3000 waiting on condition [0xb0ec2000..0xb0ec2bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04d37b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "RESPONSE-STAGE:4" prio=5 tid=0x0090bc00 nid=0xb0e71000 waiting on condition [0xb0e70000..0xb0e70858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf8b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "RESPONSE-STAGE:3" prio=5 tid=0x0090b400 nid=0xb0e1f000 waiting on condition [0xb0e1e000..0xb0e1e8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf8b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "RESPONSE-STAGE:2" prio=5 tid=0x0090a400 nid=0xb0dcd000 waiting on condition [0xb0dcc000..0xb0dcc958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf8b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "RESPONSE-STAGE:1" prio=5 tid=0x00907400 nid=0xb0d7b000 waiting on condition [0xb0d7a000..0xb0d7a9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf8b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-STREAMING-POOL:1" prio=5 tid=0x00906400 nid=0xb0d29000 waiting on condition [0xb0d28000..0xb0d28a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf1950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-DESERIALIZER-POOL:4" prio=5 tid=0x00905800 nid=0xb0cd7000 waiting on condition [0xb0cd6000..0xb0cd6ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf0bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-DESERIALIZER-POOL:3" prio=5 tid=0x00904800 nid=0xb0c85000 waiting on condition [0xb0c84000..0xb0c84b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf0bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-DESERIALIZER-POOL:2" prio=5 tid=0x00903c00 nid=0xb0c33000 waiting on condition [0xb0c32000..0xb0c32bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf0bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-DESERIALIZER-POOL:1" prio=5 tid=0x00902c00 nid=0xb0be1000 waiting on condition [0xb0be0000..0xb0be0858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cf0bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-SERIALIZER-POOL:4" prio=5 tid=0x00902000 nid=0xb0b8f000 waiting on condition [0xb0b8e000..0xb0b8e8d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ceff00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-SERIALIZER-POOL:3" prio=5 tid=0x00901000 nid=0xb0b3d000 waiting on condition [0xb0b3c000..0xb0b3c958]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ceff00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-SERIALIZER-POOL:2" prio=5 tid=0x00900400 nid=0xb0aeb000 waiting on condition [0xb0aea000..0xb0aea9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ceff00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGE-SERIALIZER-POOL:1" prio=5 tid=0x008ff400 nid=0xb0a99000 waiting on condition [0xb0a98000..0xb0a98a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04ceff00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGING-SERVICE-POOL:4" prio=5 tid=0x008fe800 nid=0xb0a47000 waiting on condition [0xb0a46000..0xb0a46ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cef1f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGING-SERVICE-POOL:3" prio=5 tid=0x008fd800 nid=0xb09f5000 waiting on condition [0xb09f4000..0xb09f4b58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cef1f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGING-SERVICE-POOL:2" prio=5 tid=0x008fd000 nid=0xb09a3000 waiting on condition [0xb09a2000..0xb09a2bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cef1f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "MESSAGING-SERVICE-POOL:1" prio=5 tid=0x008f9800 nid=0xb0951000 waiting on condition [0xb0950000..0xb0950858]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cef1f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "CACHETABLE-TIMER-2" daemon prio=5 tid=0x008fc400 nid=0xb08ff000 in Object.wait() [0xb08fe000..0xb08fe8d8]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x04ceecd0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.mainLoop(Timer.java:509)
> 	- locked <0x04ceecd0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:462)
> "CACHETABLE-TIMER-1" daemon prio=5 tid=0x008f8400 nid=0xb08ad000 in Object.wait() [0xb08ac000..0xb08ac958]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x04cee690> (a java.util.TaskQueue)
> 	at java.util.TimerThread.mainLoop(Timer.java:509)
> 	- locked <0x04cee690> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:462)
> "LOAD-BALANCER-STAGE:1" prio=5 tid=0x008f5000 nid=0xb085b000 waiting on condition [0xb085a000..0xb085a9d8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cbec18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "LB-TARGET:1" prio=5 tid=0x008f7c00 nid=0xb0809000 waiting on condition [0xb0808000..0xb0808a58]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cbb230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "BOOT-STRAPPER:1" prio=5 tid=0x008f2800 nid=0xb07b7000 waiting on condition [0xb07b6000..0xb07b6ad8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x04cace50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "Timer-0" prio=5 tid=0x008ed800 nid=0xb0765000 in Object.wait() [0xb0764000..0xb0764b58]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x04cac5e0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.mainLoop(Timer.java:509)
> 	- locked <0x04cac5e0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:462)
> "FILEUTILS-DELETE-POOL:1" prio=5 tid=0x008c4000 nid=0xb0713000 waiting on condition [0xb0712000..0xb0712bd8]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x058804f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
> 	at java.lang.Thread.run(Thread.java:619)
> "RMI TCP Accept-8080" daemon prio=5 tid=0x008a9800 nid=0xb063f000 runnable [0xb063e000..0xb063e8d8]
>    java.lang.Thread.State: RUNNABLE
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> 	- locked <0x05880740> (a java.net.SocksSocketImpl)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> 	at java.net.ServerSocket.accept(ServerSocket.java:421)
> 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> 	at java.lang.Thread.run(Thread.java:619)
> "RMI TCP Accept-0" daemon prio=5 tid=0x008a9000 nid=0xb05ed000 runnable [0xb05ec000..0xb05ec958]
>    java.lang.Thread.State: RUNNABLE
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> 	- locked <0x05880a60> (a java.net.SocksSocketImpl)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> 	at java.net.ServerSocket.accept(ServerSocket.java:421)
> 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> 	at java.lang.Thread.run(Thread.java:619)
> "Low Memory Detector" daemon prio=5 tid=0x00842800 nid=0xb059b000 runnable [0x00000000..0x00000000]
>    java.lang.Thread.State: RUNNABLE
> "CompilerThread1" daemon prio=5 tid=0x00841c00 nid=0xb0549000 waiting on condition [0x00000000..0xb0547fb8]
>    java.lang.Thread.State: RUNNABLE
> "CompilerThread0" daemon prio=5 tid=0x00840c00 nid=0xb04c7000 waiting on condition [0x00000000..0xb04c6038]
>    java.lang.Thread.State: RUNNABLE
> "JDWP Event Helper Thread" daemon prio=5 tid=0x00840000 nid=0xb0445000 runnable [0x00000000..0x00000000]
>    java.lang.Thread.State: RUNNABLE
> "JDWP Transport Listener: dt_socket" daemon prio=5 tid=0x0083e000 nid=0xb03f3000 runnable [0x00000000..0x00000000]
>    java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=5 tid=0x00837000 nid=0xb03a1000 waiting on condition [0x00000000..0x00000000]
>    java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (CMS)" daemon prio=5 tid=0x00837800 nid=0xb034f000 waiting on condition [0x00000000..0x00000000]
>    java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=5 tid=0x00827800 nid=0xb02fd000 in Object.wait() [0xb02fc000..0xb02fc958]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x0588d498> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> 	- locked <0x0588d498> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> "Reference Handler" daemon prio=5 tid=0x00827400 nid=0xb02ab000 in Object.wait() [0xb02aa000..0xb02aa9d8]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x0588d4b8> (a java.lang.ref.Reference$Lock)
> 	at java.lang.Object.wait(Object.java:485)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 	- locked <0x0588d4b8> (a java.lang.ref.Reference$Lock)
> "main" prio=5 tid=0x00800c00 nid=0xb0051000 runnable [0xb0050000..0xb0050c58]
>    java.lang.Thread.State: RUNNABLE
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> 	- locked <0x05073840> (a java.net.SocksSocketImpl)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> 	at java.net.ServerSocket.accept(ServerSocket.java:421)
> 	at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
> 	at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
> 	at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
> 	at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
> 	at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:97)
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:133)
> "VM Thread" prio=5 tid=0x00825400 nid=0xb0259000 runnable 
> "Gang worker#0 (Parallel GC Threads)" prio=5 tid=0x00801400 nid=0xb00d3000 runnable 
> "Gang worker#1 (Parallel GC Threads)" prio=5 tid=0x00805000 nid=0xb0155000 runnable 
> "Concurrent Mark-Sweep GC Thread" prio=5 tid=0x00806800 nid=0xb01d7000 runnable 
> "VM Periodic Task Thread" prio=5 tid=0x008a9c00 nid=0xb06c1000 waiting on condition 
> JNI global references: 2722
> Heap
>  par new generation   total 14784K, used 7461K [0x04ba0000, 0x05ba0000, 0x06ba0000)
>   eden space 13184K,  47% used [0x04ba0000, 0x051c7260, 0x05880000)
>   from space 1600K,  72% used [0x05880000, 0x059a2478, 0x05a10000)
>   to   space 1600K,   0% used [0x05a10000, 0x05a10000, 0x05ba0000)
>  concurrent mark-sweep generation total 114688K, used 0K [0x06ba0000, 0x0dba0000, 0x44ba0000)
>  concurrent-mark-sweep perm gen total 16384K, used 8620K [0x44ba0000, 0x45ba0000, 0x48ba0000)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.