You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Killeen, Tom" <to...@thomson.com> on 2003/08/11 16:55:12 UTC
parallel index building & searching multiple indexes
I am attempting to create approx 10 different Lucene indexes. I'm trying to
create them at the same time by running multiple processes and each index is
written to a new directory. Once I create more than one process - the
performance is very, very slow.
Any sample code out there showing an efficient way to create multiple
indexes?
Also, Any sample code out there to search the multiple indexes?
thanks,
Tom
-----Original Message-----
From: Tatu Saloranta [mailto:tatu@hypermall.net]
Sent: Monday, August 11, 2003 8:41 AM
To: Lucene Users List
Subject: Re: 2,147,483,647 max documents?
On Monday 11 August 2003 01:07, Kevin A. Burton wrote:
> Why was an int chosen to represent document handles? Is there a reason
> for this? Why wasn't a long chosen to represent document handles? 64
> bits seems like the obvious choice here except for a potentially bloated
> datastore.... (32 extra bits)
I can't speak for actual reasons (not being core Lucene developer), but the
general benefits of 32-bit ints vs. longs are:
- Better performance on pretty much any current architecture (even so-called
64-bit CPUs often prefer 32-bit data access, and 64-bit representations
are
more important for addressing).
Also, smaller data set size is usually also good for performance
(caching).
- Atomicity of access (read access can often be done without synchronizing);
longs can not be atomically accessed in Java.
Another question is whether limited address space presents a real problem.
Since Lucene can reuse doc ids (or rather, there is not persistent id per
se?
doc id is just an index, and holes left by removed docs can be reused?),
perhaps this is usually not much of an issue?
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: parallel index building & searching multiple indexes
Posted by Peter Becker <pb...@dstc.edu.au>.
Kevin A. Burton wrote:
> Killeen, Tom wrote:
>
>> I am attempting to create approx 10 different Lucene indexes. I'm
>> trying to
>> create them at the same time by running multiple processes and each
>> index is
>> written to a new directory. Once I create more than one process - the
>> performance is very, very slow.
>>
> If they are IDE disks you should NOT try to build multiple indexes at
> once since your disks will bottleneck.
>
> If you are using SCSI/Fibre disks you should be able to write out
> multiple indexes at once but you should benchmark it to make sure that
> it is in fact faster.
It is a downloadable application, so I don't know what the disks are ;-)
I guess a common scenario is that someone indexes local and network
directories. Then the parallel execution probably doesn't hurt, it might
actually help. Of course it would be smarter to index server-side, but
that is not yet implemented.
For the little application we wrote the indexing speed doesn't seem too
important anyway. And it allows updating the index which means that
benchmarking is harder -- although it probably also means that accessing
multiple locations on the same drive hurts more, since the CPU is most
likely used less in comparison to the disks if a document is up-to-date.
But I can live with it, so I rather keep it for now. Maybe one day we
change from the threaded model we have now to a queued model, but I am
not sure if our users would be happy with it, since it means they either
have to wait for the just requested update until everything else is
finished, or we would have to add controls for the processing order,
which bloats the UI.
Peter
Re: parallel index building & searching multiple indexes
Posted by Peter Becker <pb...@dstc.edu.au>.
Kevin A. Burton wrote:
> Killeen, Tom wrote:
>
>> I am attempting to create approx 10 different Lucene indexes. I'm
>> trying to
>> create them at the same time by running multiple processes and each
>> index is
>> written to a new directory. Once I create more than one process - the
>> performance is very, very slow.
>>
> If they are IDE disks you should NOT try to build multiple indexes at
> once since your disks will bottleneck.
>
> If you are using SCSI/Fibre disks you should be able to write out
> multiple indexes at once but you should benchmark it to make sure that
> it is in fact faster.
It is a downloadable application, so I don't know what the disks are ;-)
I guess a common scenario is that someone indexes local and network
directories. Then the parallel execution probably doesn't hurt, it might
actually help. Of course it would be smarter to index server-side, but
that is not yet implemented.
For the little application we wrote the indexing speed doesn't seem too
important anyway. And it allows updating the index which means that
benchmarking is harder -- although it probably also means that accessing
multiple locations on the same drive hurts more, since the CPU is most
likely used less in comparison to the disks if a document is up-to-date.
But I can live with it, so I rather keep it for now. Maybe one day we
change from the threaded model we have now to a queued model, but I am
not sure if our users would be happy with it, since it means they either
have to wait for the just requested update until everything else is
finished, or we would have to add controls for the processing order,
which bloats the UI.
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: parallel index building & searching multiple indexes
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Killeen, Tom wrote:
>I am attempting to create approx 10 different Lucene indexes. I'm trying to
>create them at the same time by running multiple processes and each index is
>written to a new directory. Once I create more than one process - the
>performance is very, very slow.
>
>
If they are IDE disks you should NOT try to build multiple indexes at
once since your disks will bottleneck.
If you are using SCSI/Fibre disks you should be able to write out
multiple indexes at once but you should benchmark it to make sure that
it is in fact faster.
Kevin
--
Help Support NewsMonster Development! Purchase NewsMonster PRO!
http://www.newsmonster.org/download-pro.html
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
AIM - sfburtonator, Web - http://www.peerfear.org/
GPG fingerprint: 4D20 40A0 C734 307E C7B4 DCAA 0303 3AC5 BD9D 7C4D
IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: parallel index building & searching multiple indexes
Posted by Peter Becker <pb...@dstc.edu.au>.
Hi Tom,
Killeen, Tom wrote:
>I am attempting to create approx 10 different Lucene indexes. I'm trying to
>create them at the same time by running multiple processes and each index is
>written to a new directory. Once I create more than one process - the
>performance is very, very slow.
>
As Otis said: disk access probably creates the trouble, since your
disk's head has to be at 20 positions at once (10 indexes, 10 input
directories). At that is assuming not too much fragmentation.
Creating them on different disks and moving them across is not a problem
as long as you don't put any relative paths in the index yourself.
Creating them sequentially instead of parallel might be another option
to speed up things.
>Any sample code out there showing an efficient way to create multiple
>indexes?
>
>Also, Any sample code out there to search the multiple indexes?
>
We actually do both -- ignoring the fact that parallel might be slower
on a single disk since we don't really know which disks are involved.
Our program is a little GUI application running the indexing in the
background (see http://tockit.sf.net/docco). You can find our code here:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/
or to be more specific:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/source/org/tockit/docco/
The bits you ask for are in the "index", "indexer" and "query" packages.
"documenthandler" might also be interesting since it contains code to
read text, html, xml and OpenOffice documents. There are also plugins
for POI, PDFbox and Multivalent here:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/plugins/
The code can be checked out via anonymous cvs as usual with Sourceforge:
http://sourceforge.net/cvs/?group_id=37081
HTH,
Peter
>
>thanks,
>Tom
>
>
[...unrelated quote...]
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: parallel index building & searching multiple indexes
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Killeen, Tom wrote:
>I am attempting to create approx 10 different Lucene indexes. I'm trying to
>create them at the same time by running multiple processes and each index is
>written to a new directory. Once I create more than one process - the
>performance is very, very slow.
>
>
If they are IDE disks you should NOT try to build multiple indexes at
once since your disks will bottleneck.
If you are using SCSI/Fibre disks you should be able to write out
multiple indexes at once but you should benchmark it to make sure that
it is in fact faster.
Kevin
--
Help Support NewsMonster Development! Purchase NewsMonster PRO!
http://www.newsmonster.org/download-pro.html
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
AIM - sfburtonator, Web - http://www.peerfear.org/
GPG fingerprint: 4D20 40A0 C734 307E C7B4 DCAA 0303 3AC5 BD9D 7C4D
IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster
Re: parallel index building & searching multiple indexes
Posted by Peter Becker <pb...@dstc.edu.au>.
Hi Tom,
Killeen, Tom wrote:
>I am attempting to create approx 10 different Lucene indexes. I'm trying to
>create them at the same time by running multiple processes and each index is
>written to a new directory. Once I create more than one process - the
>performance is very, very slow.
>
As Otis said: disk access probably creates the trouble, since your
disk's head has to be at 20 positions at once (10 indexes, 10 input
directories). At that is assuming not too much fragmentation.
Creating them on different disks and moving them across is not a problem
as long as you don't put any relative paths in the index yourself.
Creating them sequentially instead of parallel might be another option
to speed up things.
>Any sample code out there showing an efficient way to create multiple
>indexes?
>
>Also, Any sample code out there to search the multiple indexes?
>
We actually do both -- ignoring the fact that parallel might be slower
on a single disk since we don't really know which disks are involved.
Our program is a little GUI application running the indexing in the
background (see http://tockit.sf.net/docco). You can find our code here:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/
or to be more specific:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/source/org/tockit/docco/
The bits you ask for are in the "index", "indexer" and "query" packages.
"documenthandler" might also be interesting since it contains code to
read text, html, xml and OpenOffice documents. There are also plugins
for POI, PDFbox and Multivalent here:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/toscanaj/docco/plugins/
The code can be checked out via anonymous cvs as usual with Sourceforge:
http://sourceforge.net/cvs/?group_id=37081
HTH,
Peter
>
>thanks,
>Tom
>
>
[...unrelated quote...]
Re: parallel index building & searching multiple indexes
Posted by Otis Gospodnetic <ot...@yahoo.com>.
You didn't provide details about your setup, but my first guess is that
all your processes are writing the indices on the same disk, and that
disk is your bottleneck.
Otis
--- "Killeen, Tom" <to...@thomson.com> wrote:
> I am attempting to create approx 10 different Lucene indexes. I'm
> trying to
> create them at the same time by running multiple processes and each
> index is
> written to a new directory. Once I create more than one process -
> the
> performance is very, very slow.
>
> Any sample code out there showing an efficient way to create multiple
> indexes?
>
> Also, Any sample code out there to search the multiple indexes?
>
> thanks,
> Tom
>
> -----Original Message-----
> From: Tatu Saloranta [mailto:tatu@hypermall.net]
> Sent: Monday, August 11, 2003 8:41 AM
> To: Lucene Users List
> Subject: Re: 2,147,483,647 max documents?
>
>
> On Monday 11 August 2003 01:07, Kevin A. Burton wrote:
> > Why was an int chosen to represent document handles? Is there a
> reason
> > for this? Why wasn't a long chosen to represent document handles?
> 64
> > bits seems like the obvious choice here except for a potentially
> bloated
> > datastore.... (32 extra bits)
>
> I can't speak for actual reasons (not being core Lucene developer),
> but the
> general benefits of 32-bit ints vs. longs are:
>
> - Better performance on pretty much any current architecture (even
> so-called
> 64-bit CPUs often prefer 32-bit data access, and 64-bit
> representations
> are
> more important for addressing).
> Also, smaller data set size is usually also good for performance
> (caching).
> - Atomicity of access (read access can often be done without
> synchronizing);
> longs can not be atomically accessed in Java.
>
> Another question is whether limited address space presents a real
> problem.
> Since Lucene can reuse doc ids (or rather, there is not persistent id
> per
> se?
> doc id is just an index, and holes left by removed docs can be
> reused?),
> perhaps this is usually not much of an issue?
>
> -+ Tatu +-
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: parallel index building & searching multiple indexes
Posted by Otis Gospodnetic <ot...@yahoo.com>.
You didn't provide details about your setup, but my first guess is that
all your processes are writing the indices on the same disk, and that
disk is your bottleneck.
Otis
--- "Killeen, Tom" <to...@thomson.com> wrote:
> I am attempting to create approx 10 different Lucene indexes. I'm
> trying to
> create them at the same time by running multiple processes and each
> index is
> written to a new directory. Once I create more than one process -
> the
> performance is very, very slow.
>
> Any sample code out there showing an efficient way to create multiple
> indexes?
>
> Also, Any sample code out there to search the multiple indexes?
>
> thanks,
> Tom
>
> -----Original Message-----
> From: Tatu Saloranta [mailto:tatu@hypermall.net]
> Sent: Monday, August 11, 2003 8:41 AM
> To: Lucene Users List
> Subject: Re: 2,147,483,647 max documents?
>
>
> On Monday 11 August 2003 01:07, Kevin A. Burton wrote:
> > Why was an int chosen to represent document handles? Is there a
> reason
> > for this? Why wasn't a long chosen to represent document handles?
> 64
> > bits seems like the obvious choice here except for a potentially
> bloated
> > datastore.... (32 extra bits)
>
> I can't speak for actual reasons (not being core Lucene developer),
> but the
> general benefits of 32-bit ints vs. longs are:
>
> - Better performance on pretty much any current architecture (even
> so-called
> 64-bit CPUs often prefer 32-bit data access, and 64-bit
> representations
> are
> more important for addressing).
> Also, smaller data set size is usually also good for performance
> (caching).
> - Atomicity of access (read access can often be done without
> synchronizing);
> longs can not be atomically accessed in Java.
>
> Another question is whether limited address space presents a real
> problem.
> Since Lucene can reuse doc ids (or rather, there is not persistent id
> per
> se?
> doc id is just an index, and holes left by removed docs can be
> reused?),
> perhaps this is usually not much of an issue?
>
> -+ Tatu +-
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com