You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Rob Slifka (JIRA)" <ji...@apache.org> on 2010/01/26 19:48:34 UTC
[jira] Created: (THRIFT-692) RejectedExecutionException under load
in Java TThreadPoolServer
RejectedExecutionException under load in Java TThreadPoolServer
---------------------------------------------------------------
Key: THRIFT-692
URL: https://issues.apache.org/jira/browse/THRIFT-692
Project: Thrift
Issue Type: Bug
Components: Library (Java)
Environment: Linux
Reporter: Rob Slifka
Priority: Critical
Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
{code}
Starting Thrift server on 9090...
Exception in thread "main" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
at com.foo.ipc.server.ThriftServer.main(Unknown Source)
{code}
Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-692) RejectedExecutionException under
load in Java TThreadPoolServer
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805133#action_12805133 ]
David Reiss commented on THRIFT-692:
------------------------------------
>From <http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html>, "If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize, in which case, the task will be rejected." We use a SynchronousQueue, so requests can never be queued. This is the expected result when you exceed the maximum threadpool size. Please close this bug as invalid if this answers your question.
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (THRIFT-692) RejectedExecutionException under load
in Java TThreadPoolServer
Posted by "Rob Slifka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Slifka closed THRIFT-692.
-----------------------------
Resolution: Invalid
(see David's comment)
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-692) RejectedExecutionException under
load in Java TThreadPoolServer
Posted by "Rob Slifka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866269#action_12866269 ]
Rob Slifka commented on THRIFT-692:
-----------------------------------
Hi Philip,
I tend to think of this as an assert.
In this case, you're controlling both ends of the pipe: the Thrift server pool and the pool on the client side. Using Commons Pool, it's a configuration option for the size of the pool, so we never ran into this after setting them both to be the same.
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-692) RejectedExecutionException under
load in Java TThreadPoolServer
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866263#action_12866263 ]
Philip Zeyliger commented on THRIFT-692:
----------------------------------------
Not sure if it's poor form to pile onto a closed issue...
Right now, if I have a server running with, say, 10 thread serving it, if I run 11 concurrent requests against it, the server will shut down, because the exception will pass through the serve() method. I'd argue that it would be more reasonable to tell the client "go away, I'm busy", rather than triggering the server to shut down entirely.
Thoughts?
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-692) RejectedExecutionException under
load in Java TThreadPoolServer
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866282#action_12866282 ]
Philip Zeyliger commented on THRIFT-692:
----------------------------------------
Rob,
In the case where you know exactly how many clients there could be, that's sensible, but if you have many clients, you don't want them to be able to quite so easily kill the server.
-- Philip
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-692) RejectedExecutionException under
load in Java TThreadPoolServer
Posted by "Rob Slifka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844433#action_12844433 ]
Rob Slifka commented on THRIFT-692:
-----------------------------------
Yep, that's it! Thanks very much David, I will close as invalid.
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-692) RejectedExecutionException under load
in Java TThreadPoolServer
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Philip Zeyliger updated THRIFT-692:
-----------------------------------
Attachment: THRIFT-692.patch.untested.txt
FWIW, here's what I have in mind. Not a complete patch because it's missing tests and such, but, for my use case, I've forked a copy of this class with something like this patch.
> RejectedExecutionException under load in Java TThreadPoolServer
> ---------------------------------------------------------------
>
> Key: THRIFT-692
> URL: https://issues.apache.org/jira/browse/THRIFT-692
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Environment: Linux
> Reporter: Rob Slifka
> Priority: Critical
> Attachments: THRIFT-692.patch.untested.txt
>
>
> Under load, we're seeing this unpredictably. We're using a version of HEAD from August of 2009 since it's worked so well for us. The code that we're running against are essentially stubs for when we hook up to "real" Thrift server. I.e. our Thrift calls are answered by a server that returns literals without doing any "work" per se.
> {code}
> Starting Thrift server on 9090...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:185)
> at com.foo.ipc.server.ThriftServer.main(Unknown Source)
> {code}
> Ring a bell?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.