You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Lance Norskog (Created) (JIRA)" <ji...@apache.org> on 2012/03/03 03:57:57 UTC
[jira] [Created] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Allow firstSearcher and newSearcher listeners to run in multiple threads
------------------------------------------------------------------------
Key: SOLR-3197
URL: https://issues.apache.org/jira/browse/SOLR-3197
Project: Solr
Issue Type: Improvement
Reporter: Lance Norskog
SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
line 965 in the trunk:
{code}
SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
line 1280 in the trunk:
SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
{code}
java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221715#comment-13221715 ]
Yonik Seeley commented on SOLR-3197:
------------------------------------
The original rational was "do we really want to peg more than one CPU with warmup while we are still serving live traffic?"
But some people are using pretty big boxes these days.
Beware some startup code that expects a single threaded executor though - we can't just switch that w/o other code changes!
See the SolrCore constructor for more details.
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Tommaso Teofili (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281866#comment-13281866 ]
Tommaso Teofili commented on SOLR-3197:
---------------------------------------
I think, as Marks was saying, we could expose an option to define if warmups run on one or more threads, default to 1.
This would just not change things as they are now and people could just explicitly set the no. of threads for warmup.
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Neil Hooey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222790#comment-13222790 ]
Neil Hooey commented on SOLR-3197:
----------------------------------
{quote}
The original rational was "do we really want to peg more than one CPU with warmup while we are still serving live traffic?"
But some people are using pretty big boxes these days.
{quote}
When Solr is configured _not_ to serve queries while warming, would it make sense to use as many CPUs as possible, since it's probably not doing anything else?
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221813#comment-13221813 ]
Lance Norskog commented on SOLR-3197:
-------------------------------------
bq. The original rational was "do we really want to peg more than one CPU with warmup while we are still serving live traffic?" But some people are using pretty big boxes these days.
In this case 24 cores, tens of facet fields, hundreds of thousands of facets in some fields.
Those few lines of code are the only multi-threaded code in SolrCore that I am smart enough to understand.
bq. An alternative would be to use the CachedThreadPool as default as it makes it possible to reuse cached threads
This task starts N long-lived threads and waits until they all finish. I think the thread cache is useful when you have lots of short-lived threads.
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Tommaso Teofili (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221539#comment-13221539 ]
Tommaso Teofili commented on SOLR-3197:
---------------------------------------
An alternative would be to use the CachedThreadPool as default as it makes it possible to reuse cached threads (see http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool() )
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3197) Allow firstSearcher and newSearcher
listeners to run in multiple threads
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221519#comment-13221519 ]
Mark Miller commented on SOLR-3197:
-----------------------------------
Yeah, perhaps we can make it configurable, default to 1.
> Allow firstSearcher and newSearcher listeners to run in multiple threads
> ------------------------------------------------------------------------
>
> Key: SOLR-3197
> URL: https://issues.apache.org/jira/browse/SOLR-3197
> Project: Solr
> Issue Type: Improvement
> Reporter: Lance Norskog
>
> SolrCore submits all listeners (firstSearcher and newSearcher) to a java ExecutorService, but uses a single-threaded one.
> line 965 in the trunk:
> {code}
> SolrCore.java around line 965: final ExecutorService searcherExecutor = Executors.newSingleThreadExecutor();
> line 1280 in the trunk:
> SolrCore.java around line 1280 runs first the, and then the first and new searchers, all with the searcherExecutor object created at line 965.
> Would it work if we changed this ExecutorService to a thread pool version? This seems like it should work:
> {code}
> java.util.concurrent.Executors.newFixedThreadPool(int nThreads, ThreadFactory threadFactory);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org