You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Karl Wettin <ka...@gmail.com> on 2008/07/16 15:41:06 UTC
Scaling
Is there some sort of a scaling strategies listing available? I think
there is a Wiki page missing.
What are the typical promblems I'll encounter when distributing the
search over multiple machines?
Do people split up their index per node or do they use the complete
index and restrict what part to search in using filters? The latter
would be good for the scores, right? Then how do I calculate the cost
in speed for the score with better quality? I mean, splitting the
index in two and searching on two machines using ParallelMultiSearcher
probably means that I'll get something like 30% speed improvement and
not 100%. Or?
Is there something to win by using multiple threads each restricted to
a part each of the same index on a single machine, compared to a
single thread? Or is it all I/O? That would mean there is something to
gain if the index was on SSD or in RAM, right?
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scaling
Posted by Jason Rutherglen <ja...@gmail.com>.
RMI search in Lucene uses Searchable.int[] docFreqs(Term[] terms) to obtain
the docfreqs for all terms in a the query from each server. Which it then
turns into a globalized Weight that is submitted to all the Searchables
(servers). Look at MultiSearcher. This is fine for most systems even with
the network traffic.
There are ways to mitigate network traffic at the greater than 100 server
mark by locating a proxy server at each rack. This way the main search
client does not have for example 5000 sockets open to 5000 servers, it only
has 500 sockets open to the 500 proxy servers with 100 server per proxy.
The intensive local network traffic is performed by the search proxy which
then returns the results to the main search client.
In web search a global term frequency database is used. This does not work
for many Lucene instances where the the index is updated frequently. It
still could be done with indexes that are updates often however it would
seem to require a lot of work with possibly little to gain, unless you want
to offer the user 0.05 second response times.
On Fri, Jul 18, 2008 at 3:49 AM, Eric Bowman <eb...@boboco.ie> wrote:
> Jason Rutherglen wrote:
>
>> The scaling per machine should be linear. The overhead from the network
>> is
>> minimal because the Lucene object sizes are not impacting. Google
>> mentions
>> in one of their early white papers on scaling
>> http://labs.google.com/papers/googlecluster-ieee.pdf that they have sub
>> indexes which are now popularly called shards over which an individual
>> thread performs a search over. Executed in parallel
>> (ParallelMultiSearcher
>> which does not use thread pooling) the response time will be faster than
>> using a single thread assuming part of the indexes are in the system
>> cache.
>> A query is simply an iteration so it is easy to see how parallelization
>> speeds up response times. Queries per second should ideally be solved by
>> adding more hardware with the same indexes on each server. Then further
>> dividing these into what can be termed cells which represent different
>> indexes on sets of servers.
>>
>> Having a large index on a single machine does not scale well because most
>> of
>> the index will not be in the system cache. If the index grows so does the
>> response time. Dividing the index up into shards and cells allows for
>> efficient scaling which is proven at the big G. It puts more of the total
>> index in the system cache of many machines.
>>
>> The general assumption is that hardware is cheap and can be added easily,
>> search systems can take advantage of this and parallelize as much as
>> possible, per server, per application.
>>
>>
> One thing I have trouble understanding is how scoring works in this case.
> Does Lucene really "just work", or are there special things we have to do
> to make sure that the scores are coherent so we can actually decide which
> was the best match? What kind of constraints are there when breaking up the
> index into parts to make sure scoring remains coherent?
>
> Thanks,
> Eric
>
> --
> Eric Bowman
> Boboco Ltd
> ebowman@boboco.ie
> http://www.boboco.ie/ebowman/pubkey.pgp
> +35318394189/+353872801532
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Scaling
Posted by Karl Wettin <ka...@gmail.com>.
18 jul 2008 kl. 09.49 skrev Eric Bowman:
> One thing I have trouble understanding is how scoring works in this
> case. Does Lucene really "just work", or are there special things
> we have to do to make sure that the scores are coherent so we can
> actually decide which was the best match? What kind of constraints
> are there when breaking up the index into parts to make sure scoring
> remains coherent?
AFAIK the score would suffer from splitting up the index as tf/idf
then only represent a part of the index, i.e. two identical docments
in two indices would end up with different scores as the index meta
data is different. I have no clue how large the impact could be nor if
there are good and bad ways to split an index.
One solution I can think of is to share complete index over all nodes
but restrict the results from each node to a subset of the index using
a filter. This should produce the right score but will probably be a
bit slower than splitting the index.
Perhaps it would be possible to split the index for searching but use
an alternative source for scoring.
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scaling
Posted by Eric Bowman <eb...@boboco.ie>.
Jason Rutherglen wrote:
> The scaling per machine should be linear. The overhead from the network is
> minimal because the Lucene object sizes are not impacting. Google mentions
> in one of their early white papers on scaling
> http://labs.google.com/papers/googlecluster-ieee.pdf that they have sub
> indexes which are now popularly called shards over which an individual
> thread performs a search over. Executed in parallel (ParallelMultiSearcher
> which does not use thread pooling) the response time will be faster than
> using a single thread assuming part of the indexes are in the system cache.
> A query is simply an iteration so it is easy to see how parallelization
> speeds up response times. Queries per second should ideally be solved by
> adding more hardware with the same indexes on each server. Then further
> dividing these into what can be termed cells which represent different
> indexes on sets of servers.
>
> Having a large index on a single machine does not scale well because most of
> the index will not be in the system cache. If the index grows so does the
> response time. Dividing the index up into shards and cells allows for
> efficient scaling which is proven at the big G. It puts more of the total
> index in the system cache of many machines.
>
> The general assumption is that hardware is cheap and can be added easily,
> search systems can take advantage of this and parallelize as much as
> possible, per server, per application.
>
One thing I have trouble understanding is how scoring works in this
case. Does Lucene really "just work", or are there special things we
have to do to make sure that the scores are coherent so we can actually
decide which was the best match? What kind of constraints are there
when breaking up the index into parts to make sure scoring remains coherent?
Thanks,
Eric
--
Eric Bowman
Boboco Ltd
ebowman@boboco.ie
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scaling
Posted by Jason Rutherglen <ja...@gmail.com>.
The scaling per machine should be linear. The overhead from the network is
minimal because the Lucene object sizes are not impacting. Google mentions
in one of their early white papers on scaling
http://labs.google.com/papers/googlecluster-ieee.pdf that they have sub
indexes which are now popularly called shards over which an individual
thread performs a search over. Executed in parallel (ParallelMultiSearcher
which does not use thread pooling) the response time will be faster than
using a single thread assuming part of the indexes are in the system cache.
A query is simply an iteration so it is easy to see how parallelization
speeds up response times. Queries per second should ideally be solved by
adding more hardware with the same indexes on each server. Then further
dividing these into what can be termed cells which represent different
indexes on sets of servers.
Having a large index on a single machine does not scale well because most of
the index will not be in the system cache. If the index grows so does the
response time. Dividing the index up into shards and cells allows for
efficient scaling which is proven at the big G. It puts more of the total
index in the system cache of many machines.
The general assumption is that hardware is cheap and can be added easily,
search systems can take advantage of this and parallelize as much as
possible, per server, per application.
On Wed, Jul 16, 2008 at 9:41 AM, Karl Wettin <ka...@gmail.com> wrote:
> Is there some sort of a scaling strategies listing available? I think there
> is a Wiki page missing.
>
> What are the typical promblems I'll encounter when distributing the search
> over multiple machines?
>
> Do people split up their index per node or do they use the complete index
> and restrict what part to search in using filters? The latter would be good
> for the scores, right? Then how do I calculate the cost in speed for the
> score with better quality? I mean, splitting the index in two and searching
> on two machines using ParallelMultiSearcher probably means that I'll get
> something like 30% speed improvement and not 100%. Or?
>
> Is there something to win by using multiple threads each restricted to a
> part each of the same index on a single machine, compared to a single
> thread? Or is it all I/O? That would mean there is something to gain if the
> index was on SSD or in RAM, right?
>
>
> karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Scaling
Posted by Glen Newton <gl...@gmail.com>.
A subset of your questions are answered (or at least examined) in my
postings on multi-thread queries on a multiple-core single system:
http://zzzoot.blogspot.com/2008/06/simultaneous-threaded-query-lucene.html
http://zzzoot.blogspot.com/2008/06/lucene-concurrent-search-performance.html
-Glen
2008/7/16 Karl Wettin <ka...@gmail.com>:
> Is there some sort of a scaling strategies listing available? I think there
> is a Wiki page missing.
>
> What are the typical promblems I'll encounter when distributing the search
> over multiple machines?
>
> Do people split up their index per node or do they use the complete index
> and restrict what part to search in using filters? The latter would be good
> for the scores, right? Then how do I calculate the cost in speed for the
> score with better quality? I mean, splitting the index in two and searching
> on two machines using ParallelMultiSearcher probably means that I'll get
> something like 30% speed improvement and not 100%. Or?
>
> Is there something to win by using multiple threads each restricted to a
> part each of the same index on a single machine, compared to a single
> thread? Or is it all I/O? That would mean there is something to gain if the
> index was on SSD or in RAM, right?
>
>
> karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org