You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jack Repenning <jr...@collab.net> on 2011/05/31 19:44:37 UTC
Using multiple CPUs for a single document base?
Is there a way to allow Solr to use multiple CPUs of a single, multi-core box, to increase scale (number of documents, number of searches) of the searchbase?
The CoreAdmin wiki page talks about "Multiple Cores" as essentially independent document bases with independent indexes, but with some unification of administration at the grosser levels. That's not quite what I'm looking for, though. I want a single URL for add and search access, and a single logical searchbase, but I want to be able to use more of the resources of the physical box where the searchbase runs.
I guess I thought I would get this for free, it being Java and all, but I don't seem to: even with hundreds of clients adding and searching, I only seem to use one hardware core, and a bit of a second (which I interpret to mean "one Java thread for Solr, one Java thread for Java I/O").
-==-
Jack Repenning
Technologist
Codesion Business Unit
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Markus Jelsma <ma...@openindex.io>.
1.3 is the schema version. It hasn't had as many upgrades. Solr 3.1 uses the
1.3 schema version, Solr 1.4.x uses the 1.2 schema version.
> On May 31, 2011, at 11:16 AM, Markus Jelsma wrote:
> > Are you using a < 1.4 version of Solr?
>
> Yeah, about those version numbers ... The tarball I installed claimed its
> version was
>
> apache-solr-3.1.0
>
> Which sounds comfortably later than 1.4.
>
> But the examples/solr/schema.xml that comes with it claims version 1.3.
>
> I'm confused.
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jack Repenning <jr...@collab.net>.
On May 31, 2011, at 11:16 AM, Markus Jelsma wrote:
> Are you using a < 1.4 version of Solr?
Yeah, about those version numbers ... The tarball I installed claimed its version was
apache-solr-3.1.0
Which sounds comfortably later than 1.4.
But the examples/solr/schema.xml that comes with it claims version 1.3.
I'm confused.
-==-
Jack Repenning
Technologist
Codesion Business Unit
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Markus Jelsma <ma...@openindex.io>.
Are you using a < 1.4 version of Solr? It has since been improved for multi-
threaded scalability such as a lot of neat non-blocking components. It's been
a long time ago since i saw a Solr server not taking advantage of multiple
cores. Today, they take 200% up to even 1200% CPU time when viewing with top.
Here's one running at almost 700% processing 450+ queries/second.
15522 markus 20 0 2432m 859m 10m S 698 10.8 2:05.41 java
> Is there a way to allow Solr to use multiple CPUs of a single, multi-core
> box, to increase scale (number of documents, number of searches) of the
> searchbase?
>
> The CoreAdmin wiki page talks about "Multiple Cores" as essentially
> independent document bases with independent indexes, but with some
> unification of administration at the grosser levels. That's not quite what
> I'm looking for, though. I want a single URL for add and search access,
> and a single logical searchbase, but I want to be able to use more of the
> resources of the physical box where the searchbase runs.
>
> I guess I thought I would get this for free, it being Java and all, but I
> don't seem to: even with hundreds of clients adding and searching, I only
> seem to use one hardware core, and a bit of a second (which I interpret to
> mean "one Java thread for Solr, one Java thread for Java I/O").
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jack Repenning <jr...@collab.net>.
On May 31, 2011, at 12:24 PM, Jonathan Rochkind wrote:
> I do all my 'adds' to a seperate Solr index, and then replicate to a slave that actually serves queries.
Yes, that's a step I'm holding in reserve. Probably get there some day, as I expect always to have a very high add-to-query ratio. But for the moment, I don't think I need it.
> My 'master' that I do my adds to is actually on the very same server -- but I run it in an entirely different java container,
Now THAT was an interesting data point, thanks very much! I hadn't thought of running the master on the same box!
-==-
Jack Repenning
Technologist
Codesion Business Unit
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jonathan Rochkind <ro...@jhu.edu>.
Yep, that could be it.
You certainly don't get _great_ concurrency support 'for free' in Java,
concurrent programming is still tricky. Parts of Solr are surely better
at it than others.
The one place I'd be shocked was just with multiple concurrent queries,
if those weren't helped by multi-CPUs. Multi CPUs won't neccesarily
speed up any single query, they should just speed up the overall
situation under heavy load. Which has been my observation. And it may
be that multi-CPU's don't speed up add/commit much, as you possibly have
observed.
I do all my 'adds' to a seperate Solr index, and then replicate to a
slave that actually serves queries. My 'master' that I do my adds to is
actually on the very same server -- but I run it in an entirely
different java container, in part to minimize any chance that it will
end up competing for threads/CPUs with the slave serving queries, the OS
level alone should ('should', famous last word) balance it to a
different cpu core. (Of course, there's still only so much total CPU
avail on the machine).
On 5/31/2011 2:53 PM, Jack Repenning wrote:
> On May 31, 2011, at 11:29 AM, Jonathan Rochkind wrote:
>
>> I kind of think you should get multi-CPU use 'for free' as a Java app too.
> Ah, probably experimental error? If I apply a stress load consisting only of queries, I get automatic multi-core use as expected. I could see where indexing new dox could tend toward synchronization and uniprocessing. Perhaps my original test load was too add-centric, does that make sense?
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
>
>
>
>
>
>
>
>
>
Re: Using multiple CPUs for a single document base?
Posted by Erick Erickson <er...@gmail.com>.
As far as I know you're on the right track, adds are single threaded.
You can have multiple threads making indexing requests from your
client, but that's primarily aimed at making the I/O not be the bottleneck,
at some point the actual indexing of the documents is single-threaded.
It'd be tricky, very tricky to have multiple threads writing to an index at
the same time, much less multiple CPUs.....
If you're desperate to index quickly, you can index into several cores, even
on separate machines and merge the results.
Best
Erick
On Tue, May 31, 2011 at 4:13 PM, Jack Repenning <jr...@collab.net> wrote:
> On May 31, 2011, at 12:44 PM, Markus Jelsma wrote:
>
>> I haven't given it a try but perhaps opening multiple HTTP connections to the
>> update handler will end up in multiple threads thus better CPU utilization.
>
> My original test case had hundreds of HTTP connections (all to the same URL) doing adds, but seemed to use only one CPU core for adding, or to serialize the adds somehow, something like that ... at any rate, I couldn't drive CPU use above ~120% with that configuration.
>
> This is quite different from queries. For queries (or a rich query-to-add mix), I can easily drive CPU use into multiple-hundreds of % CPU, with just a few dozen concurrent query connections (running flat out). But adds resist that trick. I don't know whether this means that adds really are using a single thread, or if they're using multiple threads but synchronizing on some monitor. Actually, I can't say I care much: bottom line seems to be I only use one CPU core (plus a negligible marginal bit) for adds.
>
> Since I've confirmed that queries spread neatly, I can live with the single-thready adds. In production, it seems likely that I'll be more or less continuously spending one CPU core on adds, and the rest on queries.
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
>
>
>
>
>
>
>
>
>
>
Re: Using multiple CPUs for a single document base?
Posted by Jack Repenning <jr...@collab.net>.
On May 31, 2011, at 12:44 PM, Markus Jelsma wrote:
> I haven't given it a try but perhaps opening multiple HTTP connections to the
> update handler will end up in multiple threads thus better CPU utilization.
My original test case had hundreds of HTTP connections (all to the same URL) doing adds, but seemed to use only one CPU core for adding, or to serialize the adds somehow, something like that ... at any rate, I couldn't drive CPU use above ~120% with that configuration.
This is quite different from queries. For queries (or a rich query-to-add mix), I can easily drive CPU use into multiple-hundreds of % CPU, with just a few dozen concurrent query connections (running flat out). But adds resist that trick. I don't know whether this means that adds really are using a single thread, or if they're using multiple threads but synchronizing on some monitor. Actually, I can't say I care much: bottom line seems to be I only use one CPU core (plus a negligible marginal bit) for adds.
Since I've confirmed that queries spread neatly, I can live with the single-thready adds. In production, it seems likely that I'll be more or less continuously spending one CPU core on adds, and the rest on queries.
-==-
Jack Repenning
Technologist
Codesion Business Unit
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Markus Jelsma <ma...@openindex.io>.
I haven't given it a try but perhaps opening multiple HTTP connections to the
update handler will end up in multiple threads thus better CPU utilization.
Would be nice if someone can prove it here, i'm not in `the lab` right now ;)
> You say it like it's something you have control over; how would one
> choose to use more than one thread when indexing? I guess maybe it
> depends on how you're indexing of course; I guess if you're using SolrJ
> it's straightforward. What if you're using the ordinary HTTP Post
> interface, or DIH?
>
> On 5/31/2011 3:35 PM, Markus Jelsma wrote:
> > If you use only one thread when indexing then one one core is going to be
> > used.
> >
> >> On May 31, 2011, at 11:29 AM, Jonathan Rochkind wrote:
> >>> I kind of think you should get multi-CPU use 'for free' as a Java app
> >>> too.
> >>
> >> Ah, probably experimental error? If I apply a stress load consisting
> >> only of queries, I get automatic multi-core use as expected. I could
> >> see where indexing new dox could tend toward synchronization and
> >> uniprocessing. Perhaps my original test load was too add-centric, does
> >> that make sense?
> >>
> >> -==-
> >> Jack Repenning
> >> Technologist
> >> Codesion Business Unit
> >> CollabNet, Inc.
> >> 8000 Marina Boulevard, Suite 600
> >> Brisbane, California 94005
> >> office: +1 650.228.2562
> >> twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jonathan Rochkind <ro...@jhu.edu>.
You say it like it's something you have control over; how would one
choose to use more than one thread when indexing? I guess maybe it
depends on how you're indexing of course; I guess if you're using SolrJ
it's straightforward. What if you're using the ordinary HTTP Post
interface, or DIH?
On 5/31/2011 3:35 PM, Markus Jelsma wrote:
> If you use only one thread when indexing then one one core is going to be
> used.
>
>> On May 31, 2011, at 11:29 AM, Jonathan Rochkind wrote:
>>> I kind of think you should get multi-CPU use 'for free' as a Java app
>>> too.
>> Ah, probably experimental error? If I apply a stress load consisting only
>> of queries, I get automatic multi-core use as expected. I could see where
>> indexing new dox could tend toward synchronization and uniprocessing.
>> Perhaps my original test load was too add-centric, does that make sense?
>>
>> -==-
>> Jack Repenning
>> Technologist
>> Codesion Business Unit
>> CollabNet, Inc.
>> 8000 Marina Boulevard, Suite 600
>> Brisbane, California 94005
>> office: +1 650.228.2562
>> twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Markus Jelsma <ma...@openindex.io>.
If you use only one thread when indexing then one one core is going to be
used.
> On May 31, 2011, at 11:29 AM, Jonathan Rochkind wrote:
> > I kind of think you should get multi-CPU use 'for free' as a Java app
> > too.
>
> Ah, probably experimental error? If I apply a stress load consisting only
> of queries, I get automatic multi-core use as expected. I could see where
> indexing new dox could tend toward synchronization and uniprocessing.
> Perhaps my original test load was too add-centric, does that make sense?
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jack Repenning <jr...@collab.net>.
On May 31, 2011, at 11:29 AM, Jonathan Rochkind wrote:
> I kind of think you should get multi-CPU use 'for free' as a Java app too.
Ah, probably experimental error? If I apply a stress load consisting only of queries, I get automatic multi-core use as expected. I could see where indexing new dox could tend toward synchronization and uniprocessing. Perhaps my original test load was too add-centric, does that make sense?
-==-
Jack Repenning
Technologist
Codesion Business Unit
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
Re: Using multiple CPUs for a single document base?
Posted by Jonathan Rochkind <ro...@jhu.edu>.
Yeah, ignore the 'multiple cores' you are seeing in the docs there,
that's about something else unrelated to CPU's, as you discovered, has
nothing to do with what you're asking about, put it out of your mind.
I kind of think you should get multi-CPU use 'for free' as a Java app
too. It does for me, heavy Solr usage, I look at my stats, multiple CPU
cores are being excersized. (Of course, this is never going to be
perfectly efficient, you aren't going to get double performance by
doubling the CPUs in one box, there are various bottlenecks, as we all
know).
There are also some Java GC tuning you want to do with multiple cores,
the default JVM settings aren't usually appropriate. (You want to
background thread your GC, I forget the magic JVM invocations). But
that's probably not related to your issue if you don't even see more
than one CPU being exersized at all, weird.
On 5/31/2011 1:44 PM, Jack Repenning wrote:
> Is there a way to allow Solr to use multiple CPUs of a single, multi-core box, to increase scale (number of documents, number of searches) of the searchbase?
>
> The CoreAdmin wiki page talks about "Multiple Cores" as essentially independent document bases with independent indexes, but with some unification of administration at the grosser levels. That's not quite what I'm looking for, though. I want a single URL for add and search access, and a single logical searchbase, but I want to be able to use more of the resources of the physical box where the searchbase runs.
>
> I guess I thought I would get this for free, it being Java and all, but I don't seem to: even with hundreds of clients adding and searching, I only seem to use one hardware core, and a bit of a second (which I interpret to mean "one Java thread for Solr, one Java thread for Java I/O").
>
> -==-
> Jack Repenning
> Technologist
> Codesion Business Unit
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
>
>
>
>
>
>
>
>
>