java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down

[Waqas Badar <wa...@panasiangroup.com>]

I have ring of three nodes. When i try to decommission one of the
machine then i got the following exception:


java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
	at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor
$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
	at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106)
	at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
ERROR [Thread-11] 2010-08-04 23:38:56,140 CassandraDaemon.java (line 83)
Uncaught exception in thread Thread[Thread-11,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
	at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor
$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
	at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106)
	at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)


My Cassandra version is 0.6.3 and Operating system is Ubuntu 8.04
Server. 

Re: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down

[Peter Schuller <pe...@infidyne.com>]

> I have ring of three nodes. When i try to decommission one of the machine
> then i got the following exception:

>From reading the code I believe this is expected (though not sure
whether to call it a bug?). Decommision ends up calling
MessagingService' shutdown() which closes the socket and then shuts
down the executor. While that prevents new connections from being
accepted after the shutdown, I don't see any attempt to prevent a
recently created IncomingTcpConnection from trying to submit a
callable to the already shut down deserialization executor.

Perhaps a try/catch is in order that would log the incident with an
explanation rather than cause exception leakage that gives the
impression of something being wrong. Thoughts?

-- 
/ Peter Schuller