You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Don Werve <do...@madwombat.com> on 2010/03/26 03:13:55 UTC
RejectedExecutionException when searching with DirectSolrConnection
This is one of those fantastically irritating bugs that only pops up...
sometimes. I'm using DirectSolrConnection to provide search for a JRuby
application, and everything works great, except every now and then, I get
a ava.util.concurrent.RejectedExecutionException in the error log.
I wrote a test harness to try and duplicate the problem, but banging against
DirectSolrCollection with 20 concurrent threads, at 10k requests per thread,
just showed that Solr is... well, fast. :)
So, it's a very intermittent error, and I can't reproduce it. Anybody else
running into this?
ERROR - java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
at
java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1178)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:913)
at
org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:209)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:139)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:467)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:319)
at
org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:60)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:187)
at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:210)
at
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:185)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:187)
at org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)
at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:173)
at
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:169)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
at $_dot_.test_minus_solrize.block_8$RUBY$__block__(test-solrize.rb:87)
at
$_dot_.test_minus_solrizeBlockCallback$block_8$RUBY$__block__xx1.call(Unknown
Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:105)
at org.jruby.runtime.Block.yield(Block.java:194)
at org.jruby.RubyFixnum.times(RubyFixnum.java:259)
at org.jruby.RubyInteger.times19(RubyInteger.java:225)
at
org.jruby.RubyInteger$i_method_0_0$RUBYFRAMEDINVOKER$times19.call(org/jruby/RubyInteger$i_method_0_0$RUBYFRAMEDINVOKER$times19.gen)
at
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:116)
at
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:133)
at $_dot_.test_minus_solrize.block_7$RUBY$__block__(test-solrize.rb:83)
at
$_dot_.test_minus_solrizeBlockCallback$block_7$RUBY$__block__xx1.call(Unknown
Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:125)
at org.jruby.runtime.BlockBody.call(BlockBody.java:72)
at org.jruby.runtime.BlockBody.call(BlockBody.java:78)
at org.jruby.runtime.Block.call(Block.java:89)
at org.jruby.RubyProc.call(RubyProc.java:221)
at org.jruby.RubyProc.call(RubyProc.java:204)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:94)
at java.lang.Thread.run(Thread.java:637)
Re: RejectedExecutionException when searching with
DirectSolrConnection
Posted by Don Werve <do...@madwombat.com>.
A followup: I discovered something interesting. If I don't run Jetty in the
same JVM as DirectSolrConnection, all is well.
Nrr.
Re: RejectedExecutionException when searching with
DirectSolrConnection
Posted by Don Werve <do...@madwombat.com>.
Nope; in fact, I thought that might be the problem, so I spent some time to
make sure that I was (a) only loading a given core once, and (b) that all of
my setup/teardown was threadsafe.
So it's just one DirectSolrConnection for the lifetime of the application,
and it only gets closed when the JVM exits.
Re: RejectedExecutionException when searching with DirectSolrConnection
Posted by Mark Miller <ma...@gmail.com>.
This type of thing has come up in the past - check the searchable lists
(eg search.lucidimagination.com). Not sure there is much helpful there
though.
To be honest, this should *really* not be possible unless the executor
is in the SHUTDOWN state - which should really not be possible unless
close was called on the core.
The executor starts in the RUNNING state, and uses an unbounded task
queue - meaning it should only reject after shutting down.
Are you doing any type of core reloading? Anything else of note?
--
- Mark
http://www.lucidimagination.com
On 03/25/2010 10:13 PM, Don Werve wrote:
> This is one of those fantastically irritating bugs that only pops up...
> sometimes. I'm using DirectSolrConnection to provide search for a JRuby
> application, and everything works great, except every now and then, I get
> a ava.util.concurrent.RejectedExecutionException in the error log.
>
> I wrote a test harness to try and duplicate the problem, but banging against
> DirectSolrCollection with 20 concurrent threads, at 10k requests per thread,
> just showed that Solr is... well, fast. :)
>
> So, it's a very intermittent error, and I can't reproduce it. Anybody else
> running into this?
>
> ERROR - java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
> at
> java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1178)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:913)
> at
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:209)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:139)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
> org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:467)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:319)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:60)
> at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:187)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:210)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:185)
> at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:187)
> at org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)
> at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:173)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:169)
> at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
> at $_dot_.test_minus_solrize.block_8$RUBY$__block__(test-solrize.rb:87)
> at
> $_dot_.test_minus_solrizeBlockCallback$block_8$RUBY$__block__xx1.call(Unknown
> Source)
> at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:105)
> at org.jruby.runtime.Block.yield(Block.java:194)
> at org.jruby.RubyFixnum.times(RubyFixnum.java:259)
> at org.jruby.RubyInteger.times19(RubyInteger.java:225)
> at
> org.jruby.RubyInteger$i_method_0_0$RUBYFRAMEDINVOKER$times19.call(org/jruby/RubyInteger$i_method_0_0$RUBYFRAMEDINVOKER$times19.gen)
> at
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:116)
> at
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:133)
> at $_dot_.test_minus_solrize.block_7$RUBY$__block__(test-solrize.rb:83)
> at
> $_dot_.test_minus_solrizeBlockCallback$block_7$RUBY$__block__xx1.call(Unknown
> Source)
> at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:125)
> at org.jruby.runtime.BlockBody.call(BlockBody.java:72)
> at org.jruby.runtime.BlockBody.call(BlockBody.java:78)
> at org.jruby.runtime.Block.call(Block.java:89)
> at org.jruby.RubyProc.call(RubyProc.java:221)
> at org.jruby.RubyProc.call(RubyProc.java:204)
> at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:94)
> at java.lang.Thread.run(Thread.java:637)
>
>