ParallelMultiSearcher reimplementation

[Gus Holcomb <gh...@cymfony.com>]

Hello everyone,
  We are currently using Lucene 1.9.1 at work. Using a profiler, I
discovered that searching with a HitCollector in a ParallelMultiSearcher
is single threaded. By extending ParallelMultiSearcher I was able to
parallelize it without a problem (and without requiring a new lucene jar
for deployment). In addition, I re-implemented all of the existing
multithreading using a user configurable thread pool, queue and executor
service, etc. The current implementation of spawning one thread per
searchable is not only slower, but dangerous.

Is this development already taking place in the trunk? I was unable to
uncover any progress in this area. I haven't contributed to lucene (or
any open source project) before, but I would be willing to clean up a
number of things in this area if there was interest.

Looking forward to hearing from you...

Gus Holcomb

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: ParallelMultiSearcher reimplementation

[Chris Hostetter <ho...@fucit.org>]

Hi Gus,

: Is this development already taking place in the trunk? I was unable to
: uncover any progress in this area. I haven't contributed to lucene (or
: any open source project) before, but I would be willing to clean up a
: number of things in this area if there was interest.

I'm not aware of any particular development going on with
ParallelMultiSearcher at the moment, but I'm also not that familiar with
ParallelMultiSearcher in general -- so I can't say for sure.

The best way to contribute these cahnges back to the community is using
the steps outlined here...

	http://wiki.apache.org/jakarta-lucene/HowToContribute

In a nut shell:

  1) check out the trunk using anon SVN
  2) move your code into the lucene package structure
  3) generate a patch using (svn add as neccessary and) svn diff
  4) submit the patch as an attachment in Jira.


Don't worry too much about "cleaning" up the code at first ... as long as
it compiles i would submit a patch, that way you can get eyeballs on it
sooner, and then worry about how clean it is.

-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: ParallelMultiSearcher reimplementation

[Doug Cutting <cu...@apache.org>]

Gus Holcomb wrote:
>   We are currently using Lucene 1.9.1 at work. Using a profiler, I
> discovered that searching with a HitCollector in a ParallelMultiSearcher
> is single threaded. By extending ParallelMultiSearcher I was able to
> parallelize it without a problem (and without requiring a new lucene jar
> for deployment). In addition, I re-implemented all of the existing
> multithreading using a user configurable thread pool, queue and executor
> service, etc. The current implementation of spawning one thread per
> searchable is not only slower, but dangerous.

Please consider breaking these into separate patches, one to permit 
ParallelMultiSearcher w/ HitCollector to not be single-threaded, and 
another to re-implement things with a thread pool.  The latter is more 
controversial, and it would be a shame to have the former wait on it.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org