You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Vico Marziale <vi...@gmail.com> on 2009/06/10 01:17:58 UTC
Common Bottlenecks
Hello all. I am new to Lucene as well as this list. I am a PhD student at
the University of New Orleans. My current research in in leveraging
highly-multicore processors to speed computer forensics tools. For the
moment I am trying to figure out what the most common performance bottleneck
inside of Lucene itself is. I will then take a crack at porting some (small)
portion of Lucene to CUDA (http://www.nvidia.com/object/cuda_what_is.html)
and see what kind of speedups are achievable.
The portion of code to be ported must be trivially parallelizable. After
spending some time digging around the docs and source, StandardAnalyzer
appears to be a likely candidate. I've run the demo code through a profiler,
but it was less than helpful, especially in light of the fact bottlenecks
are going to be dependent on the way the Lucene API is used. In
general, what is the most computationally expensive part of the process?
Does the analyzer seem like a reasonable choice?
Thanks,
--
Lodovico Marziale
PhD Candidate
Department of Computer Science
University of New Orleans
Re: Common Bottlenecks
Posted by Chris Hostetter <ho...@fucit.org>.
On Tue, 9 Jun 2009, Vico Marziale wrote:
: highly-multicore processors to speed computer forensics tools. For the
: moment I am trying to figure out what the most common performance bottleneck
: inside of Lucene itself is. I will then take a crack at porting some (small)
: portion of Lucene to CUDA (http://www.nvidia.com/object/cuda_what_is.html)
: and see what kind of speedups are achievable.
...
: appears to be a likely candidate. I've run the demo code through a profiler,
: but it was less than helpful, especially in light of the fact bottlenecks
: are going to be dependent on the way the Lucene API is used. In
: general, what is the most computationally expensive part of the process?
Vico: it doesn't look like you got any replies to your question.
performance isn't something i generally focus on when working on lucene,
but my suggestion for finding hot spots that could be improved is to look
at the benchmark tests in the contrib/benchmark directory.
Running some of those in a profiler should help you spot the likely
candidates for improvements when dealing with non-trivial usecases.
one thing to keep in mind is that search performance tends to be
completley seperate form indexing performance ... you may want to tackle
just one of those types of code paths. search tends to be the type of task
that people are most concerned with optimizing for speed.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Common Bottlenecks
Posted by Peter Keegan <pe...@gmail.com>.
I just want to point out that I don't consider these bottlenecks to be an
issue. This is exactly where I'd expect to see hotspots from the profiler,
and the overall performance of Lucene search is still stunning.
Peter
On Wed, Jun 24, 2009 at 10:13 AM, Peter Keegan <pe...@gmail.com>wrote:
> Our biggest bottleneck in searching is in a custom scorer which calls
> AllTermDocs.next() very frequently. This class uses Lucene's own BitVector,
> which I think is already highly optimized. Farther down in the list are
> DocSetHitCollector.collect() and FieldSortedQueue.insert(). For indexing,
> the main bottlneck is in the Analyzer/Filter, which is basically a
> WhitespaceAnalyzer with custom code to add payloads to tokens and change the
> positions between tokens.
>
>
> Peter
>
>
>
> On Tue, Jun 9, 2009 at 7:17 PM, Vico Marziale <vi...@gmail.com> wrote:
>
>> Hello all. I am new to Lucene as well as this list. I am a PhD student at
>> the University of New Orleans. My current research in in leveraging
>> highly-multicore processors to speed computer forensics tools. For the
>> moment I am trying to figure out what the most common performance bottleneck
>> inside of Lucene itself is. I will then take a crack at porting some (small)
>> portion of Lucene to CUDA (http://www.nvidia.com/object/cuda_what_is.html)
>> and see what kind of speedups are achievable.
>>
>> The portion of code to be ported must be trivially parallelizable. After
>> spending some time digging around the docs and source, StandardAnalyzer
>> appears to be a likely candidate. I've run the demo code through a profiler,
>> but it was less than helpful, especially in light of the fact bottlenecks
>> are going to be dependent on the way the Lucene API is used. In
>> general, what is the most computationally expensive part of the process?
>> Does the analyzer seem like a reasonable choice?
>>
>> Thanks,
>> --
>> Lodovico Marziale
>> PhD Candidate
>> Department of Computer Science
>> University of New Orleans
>>
>
>
Re: Common Bottlenecks
Posted by Peter Keegan <pe...@gmail.com>.
Our biggest bottleneck in searching is in a custom scorer which calls
AllTermDocs.next() very frequently. This class uses Lucene's own BitVector,
which I think is already highly optimized. Farther down in the list are
DocSetHitCollector.collect() and FieldSortedQueue.insert(). For indexing,
the main bottlneck is in the Analyzer/Filter, which is basically a
WhitespaceAnalyzer with custom code to add payloads to tokens and change the
positions between tokens.
Peter
On Tue, Jun 9, 2009 at 7:17 PM, Vico Marziale <vi...@gmail.com> wrote:
> Hello all. I am new to Lucene as well as this list. I am a PhD student at
> the University of New Orleans. My current research in in leveraging
> highly-multicore processors to speed computer forensics tools. For the
> moment I am trying to figure out what the most common performance bottleneck
> inside of Lucene itself is. I will then take a crack at porting some (small)
> portion of Lucene to CUDA (http://www.nvidia.com/object/cuda_what_is.html)
> and see what kind of speedups are achievable.
>
> The portion of code to be ported must be trivially parallelizable. After
> spending some time digging around the docs and source, StandardAnalyzer
> appears to be a likely candidate. I've run the demo code through a profiler,
> but it was less than helpful, especially in light of the fact bottlenecks
> are going to be dependent on the way the Lucene API is used. In
> general, what is the most computationally expensive part of the process?
> Does the analyzer seem like a reasonable choice?
>
> Thanks,
> --
> Lodovico Marziale
> PhD Candidate
> Department of Computer Science
> University of New Orleans
>