You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Teruhiko Kurosaka <Ku...@basistech.com> on 2009/10/26 19:19:50 UTC
Solr 1.4 (RC) performance on multi-CPU system
Is Solr 1.4 (Release Candidate) suppose to take advantage
of muti-CPU (core) system? I.e. if more than one update or
search requests come in about the same time, they can be
automatically assigned to differnt CPUs if available
(and the OS does its job right)?
BTW, the term "multicore" in Solr discussion is about having
multiple indexes, and has little to do with a system having more
than one CPU cores, doesn't it?
-kuro
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Oct 28, 2009 at 6:38 PM, gabriele renzi <rf...@gmail.com> wrote:
> While I'm at it, FindBugs has a few issues with the solr code base (a
> few hundreds, but most seem false positives) but it does have a few
> concurrency related issues that appear reasonable to me, is it of any
> use if I submit patches for those too?
Sure... they may or may not be bugs, but it might be at least good to
comment them (at a minimum) so others won't have to try and figure it
out.
-Yonik
http://www.lucidimagination.com
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by gabriele renzi <rf...@gmail.com>.
2009/10/28 gabriele renzi <rf...@gmail.com>:
> On Tue, Oct 27, 2009 at 3:41 PM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>
>>> is there a reason why StreamingUpdateSolrServer does not accept a
>>> connection manager argument as CommonsHttpSolrServer does?
>>
>> I don't see any reason why it shouldn't... can you open a JIRA issue
>> and optionally provide a patch?
>
> great, will do both later today (GMT+1 lunch time :) )
done, I hope I have not done anything stupid. No new tests attached
but old ones still pass and testing an inherited ctor seems moot.
While I'm at it, FindBugs has a few issues with the solr code base (a
few hundreds, but most seem false positives) but it does have a few
concurrency related issues that appear reasonable to me, is it of any
use if I submit patches for those too?
--
blog en: http://www.riffraff.info
blog it: http://riffraff.blogsome.com
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by gabriele renzi <rf...@gmail.com>.
On Tue, Oct 27, 2009 at 3:41 PM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
>> is there a reason why StreamingUpdateSolrServer does not accept a
>> connection manager argument as CommonsHttpSolrServer does?
>
> I don't see any reason why it shouldn't... can you open a JIRA issue
> and optionally provide a patch?
great, will do both later today (GMT+1 lunch time :) )
>> Its buffering feature seems kind of disjoint from the self-management
>> of a connection pool
>
> Does this just refer to the missing constructor allowing for
> HttpClient to be passed, or is there another potential improvement you
> see?
the former, I just meant I did not see any dependency between the two
that forbid allowing custom managers.
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Oct 27, 2009 at 10:23 AM, gabriele renzi <rf...@gmail.com> wrote:
> On Mon, Oct 26, 2009 at 10:43 PM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>> On Mon, Oct 26, 2009 at 4:32 PM, Teruhiko Kurosaka <Ku...@basistech.com> wrote:
>>> Are multiple CPUs utilized at indexing time as well, or just by searcher?
>>
>> Yes, multiple CPUs are utilized for indexing.
>>
>> If you're using SolrJ, and easy way to exploit this parallelism is to use
>> http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html
>
> is there a reason why StreamingUpdateSolrServer does not accept a
> connection manager argument as CommonsHttpSolrServer does?
I don't see any reason why it shouldn't... can you open a JIRA issue
and optionally provide a patch?
> Its buffering feature seems kind of disjoint from the self-management
> of a connection pool
Does this just refer to the missing constructor allowing for
HttpClient to be passed, or is there another potential improvement you
see?
-Yonik
http://www.lucidimagination.com
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by gabriele renzi <rf...@gmail.com>.
On Mon, Oct 26, 2009 at 10:43 PM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> On Mon, Oct 26, 2009 at 4:32 PM, Teruhiko Kurosaka <Ku...@basistech.com> wrote:
>> Are multiple CPUs utilized at indexing time as well, or just by searcher?
>
> Yes, multiple CPUs are utilized for indexing.
>
> If you're using SolrJ, and easy way to exploit this parallelism is to use
> http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html
is there a reason why StreamingUpdateSolrServer does not accept a
connection manager argument as CommonsHttpSolrServer does?
Its buffering feature seems kind of disjoint from the self-management
of a connection pool, and source-wise it seems to me that it relies on
the superclass' getHttpClient() but I'm probably missing something :).
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Oct 26, 2009 at 4:32 PM, Teruhiko Kurosaka <Ku...@basistech.com> wrote:
> Are multiple CPUs utilized at indexing time as well, or just by searcher?
Yes, multiple CPUs are utilized for indexing.
If you're using SolrJ, and easy way to exploit this parallelism is to use
http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html
-Yonik
http://www.lucidimagination.com
RE: Solr 1.4 (RC) performance on multi-CPU system
Posted by Teruhiko Kurosaka <Ku...@basistech.com>.
Thank you, Yonik.
Are multiple CPUs utilized at indexing time as well, or just by searcher?
-kuro
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
> Of Yonik Seeley
> Sent: Monday, October 26, 2009 11:27 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr 1.4 (RC) performance on multi-CPU system
>
> 2009/10/26 Teruhiko Kurosaka <Ku...@basistech.com>:
> > Is Solr 1.4 (Release Candidate) suppose to take advantage
> of muti-CPU
> > (core) system? I.e. if more than one update or search
> requests come in
> > about the same time, they can be automatically assigned to differnt
> > CPUs if available (and the OS does its job right)?
>
> Yes.
> Solr 1.4 will be more efficient than 1.3 for concurrent requests too:
> http://yonik.wordpress.com/2008/12/01/solr-scalability-improvements/
>
> > BTW, the term "multicore" in Solr discussion is about
> having multiple
> > indexes, and has little to do with a system having more
> than one CPU
> > cores, doesn't it?
>
> Right.
>
> -Yonik
> http://www.lucidimagination.com
>
Re: Solr 1.4 (RC) performance on multi-CPU system
Posted by Yonik Seeley <yo...@lucidimagination.com>.
2009/10/26 Teruhiko Kurosaka <Ku...@basistech.com>:
> Is Solr 1.4 (Release Candidate) suppose to take advantage
> of muti-CPU (core) system? I.e. if more than one update or
> search requests come in about the same time, they can be
> automatically assigned to differnt CPUs if available
> (and the OS does its job right)?
Yes.
Solr 1.4 will be more efficient than 1.3 for concurrent requests too:
http://yonik.wordpress.com/2008/12/01/solr-scalability-improvements/
> BTW, the term "multicore" in Solr discussion is about having
> multiple indexes, and has little to do with a system having more
> than one CPU cores, doesn't it?
Right.
-Yonik
http://www.lucidimagination.com