You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by wu fox <fo...@gmail.com> on 2006/05/21 15:02:50 UTC

Fwd: How to combine results from several indices

---------- Forwarded message ----------
From: wu fox <fo...@gmail.com>
Date: 2006-5-21 下午8:58
Subject: How to combine results from several indices
To: java-dev-faq@lucene.apache.org


Hi all:
  I have several indices and each index describe part of a document.
For example ,a index contains dublin core information of a document,
and another index contains some classification information of the same
document. If I have a query "title:lucene AND classification:project",
First how can I adjust the query so that I can search title on the
first index and classification on the second index. Second each doc of
search results contains a field "uuid",so I need to combine the docs
of the two search results which contain the same uuid. The final
result contains docs satisfy the original query. Can anybody help me?
thank you
--
爱生活,爱FOX


-- 
爱生活,爱FOX

Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
Wu,

I don't know of a general solution that you could get now.

I do have such a solution, but it is part of a much larger mechanism. 
My Company will likely authorize contribution of this code if there is
sufficient community interest in using and enhancing it.  I plan to
describe the capabilities and assess that sometime soon.  So, if you are
still looking for a solution then, we may have something for you.

Sorry about the timing,

Chuck

wu fox wrote on 05/21/2006 05:09 PM:
> Thank you chuck !!! That really helps me.By the way ,Is there a good
> algorithm to meet the constraints of ParallelReader,or some others
> have implemented some code for it? Thank you
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
Wu,

Glad to hear that!  Congratulations on getting it working.  Looking
forward to your contribution,

Chuck

wu fox wrote on 06/16/2006 03:30 PM:
> Hi ,chuck. I have implment my own parallelReader by override methods like
> Document and ParallelTermDocs ,and it really works.Your idea isnpired
> me and
> I highly appreciate you help.Maybe after some bug fix I can contribute my
> code so that everyone can share the idea and implementation if they
> encounter similar problen
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
Hi ,chuck. I have implment my own parallelReader by override methods like
Document and ParallelTermDocs ,and it really works.Your idea isnpired me and
I highly appreciate you help.Maybe after some bug fix I can contribute my
code so that everyone can share the idea and implementation if they
encounter similar problen

Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
You can try that approach, but I think you will find it more difficult. 
E.g., all of the primitive query classes are written specifically to use
doc-ids.  So, you either need to do you searches separately on each
subindex and then write your own routine to join the results, or you
would need to rewrite all the queries.

I use two different indexing combining techniques:

   1. ParallelReader/ParallelWriter for performance reasons in various
      circumstances; e.g., fast access to frequently used fields (in
      combination with lazy fields -- very useful for fast categorical
      analysis of large samples), fast bulk updates of mutable fields by
      copying a much smaller subindex, etc.
   2. Subindex query rewriting for accessing different types of objects
      in separate indices.  A query on the main index may contain a
      subquery that retrieves objects in a different index and rewrites
      itself into a disjunction of the uid's of those objects.  This
      approach works well assuming you can arrange indexing of fields in
      the main index with subindex uid values, and the disjunction
      expansions are not too large.

Maybe approach 2 is more what you need?  It's pretty simple to do. 
E.g., take a look at MultiTermQuery for a non-primitive query that
rewrites itself dependent on the index.  You need a similar class that
rewrites itself dependent on a different index.

Chuck


wu fox wrote on 06/13/2006 02:18 AM:
> thank you very much Chuck.But I still wondered is there any way that I
> can
> revise ParallelReader so that it do not need the same doc id
> .Can IndexReader comebine different doc according some mapping rules ?for
> example I can override Document method that combine docs from indices
> acoording to same uuid or override some other methods,I think it is much
> easier to do than a writer:) Thank you for your help again.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
thank you very much Chuck.But I still wondered is there any way that I can
revise ParallelReader so that it do not need the same doc id
.Can IndexReader comebine different doc according some mapping rules ?for
example I can override Document method that combine docs from indices
acoording to same uuid or override some other methods,I think it is much
easier to do than a writer:) Thank you for your help again.

Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
Wu,

I've contributed a version of ParallelWriter that takes a middle
ground.  ParallelWriter.addDocument() is synchronized, but the
underlying sub-index writes are done in parallel.  It would be possible
to allow ParallelWriter.addDocument() itself to be multi-threaded, but
the synchronization and recovery get more complex.  The basic idea would
be to have a thread with a work queue for each sub-index.  I may look at
this later, or if you enhance this, please submit your version.

Hope this helps,

Chuck


Chuck Williams wrote on 06/12/2006 09:05 AM:
> Hi Wu,
>
> The simplest solution is to synchronize calls to a
> ParallelWriter.addDocument() method that calls IndexWriter.addDocument()
> for each sub-index.  This will work assuming there are no exceptions and
> assuming you never refresh your IndexReader within
> ParallelWriter.addDocument().  If exceptions occur writing one of the
> sub-indexes, then you need to recover them.  The best approach I've
> found is to delete the unequal final subdocuments and optimize all the
> subindexes to restore equal doc ids.
>
> This approach has the consequence of single-threading all index
> writing.  I'm working on a solution to avoid this, but it may require
> deeper integration into the higher level IndexManager mechanism (which
> does reader reopening, journaling, recovery, and a lot of other things).
>
> If you can get by with single threading, I have a ParallelWriter class
> now that I could contribute.  If not, I'm considering the more general
> solution now, but will only be able to contribute it if it can be kept
> separate from the much larger IndexManager mechanism (which is more
> specific to my app and thus not likely a fit for your app anyway).
>
> Chuck
>
>
> wu fox wrote on 06/12/2006 02:43 AM:
>   
>> Hi Chuck:
>>  I am still looking forward to a solution which ensure to to meet the
>> constraints of
>> ParallelReader so that I can use it for my seach programm. I have
>> tried a lot of methods but none of them
>> is good enough for me because of obvious
>> bugs. Can you help me? thanks in advance
>>
>>     
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>   



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
Hi Wu,

The simplest solution is to synchronize calls to a
ParallelWriter.addDocument() method that calls IndexWriter.addDocument()
for each sub-index.  This will work assuming there are no exceptions and
assuming you never refresh your IndexReader within
ParallelWriter.addDocument().  If exceptions occur writing one of the
sub-indexes, then you need to recover them.  The best approach I've
found is to delete the unequal final subdocuments and optimize all the
subindexes to restore equal doc ids.

This approach has the consequence of single-threading all index
writing.  I'm working on a solution to avoid this, but it may require
deeper integration into the higher level IndexManager mechanism (which
does reader reopening, journaling, recovery, and a lot of other things).

If you can get by with single threading, I have a ParallelWriter class
now that I could contribute.  If not, I'm considering the more general
solution now, but will only be able to contribute it if it can be kept
separate from the much larger IndexManager mechanism (which is more
specific to my app and thus not likely a fit for your app anyway).

Chuck


wu fox wrote on 06/12/2006 02:43 AM:
> Hi Chuck:
>  I am still looking forward to a solution which ensure to to meet the
> constraints of
> ParallelReader so that I can use it for my seach programm. I have
> tried a lot of methods but none of them
> is good enough for me because of obvious
> bugs. Can you help me? thanks in advance
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
Hi Chuck:
  I am still looking forward to a solution which ensure to to meet the
constraints of
ParallelReader so that I can use it for my seach programm. I have
tried a lot of methods but none of them
is good enough for me because of obvious
bugs. Can you help me? thanks in advance

Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
I greatly appreciate your help Chuck .Thanks again :)

Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
I think it would be hard as there are many places in Lucene where it is
assumed that doc-ids uniquely identify a document.  This is the main
reason that ParallelReader has these constraints.

Are you sure the transactionalizing updates to sub-indexes will cause
you performance problems?  My application is also time-sensitive and
this doesn't seem to be a big problem, at least at the moment.  You
might try it first and see whether or not the performance is acceptable.

Chuck

wu fox wrote on 05/21/2006 05:17 PM:
> Some of the indices are time-sensitive,so a transaction may not suit
> for me. Can some method meet the time requirement besides the
> constraints of parallelReader?Thank you again
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
Some of the indices are time-sensitive,so a transaction may not suit
for me. Can some method meet the time requirement besides the
constraints of parallelReader?Thank you again

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by wu fox <fo...@gmail.com>.
Thank you chuck !!! That really helps me.By the way ,Is there a good
algorithm to meet the constraints of ParallelReader,or some others
have implemented some code for it? Thank you

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Fwd: How to combine results from several indices

Posted by Chuck Williams <ch...@manawiz.com>.
wu fox wrote on 05/21/2006 03:02 AM:
> I have several indices and each index describe part of a document.
> For example ,a index contains dublin core information of a document,
> and another index contains some classification information of the same
> document. If I have a query "title:lucene AND classification:project",
> First how can I adjust the query so that I can search title on the
> first index and classification on the second index. Second each doc of
> search results contains a field "uuid",so I need to combine the docs
> of the two search results which contain the same uuid. The final
> result contains docs satisfy the original query. Can anybody help me?

If your indices meet the constraints of ParallelReader then that could
be your answer. You can only use this if the doc-id's in the different
indices are maintained such that documents with the same uuid always
have the same doc-id. This is trickier than it might sound. E.g., if you
have multi-threaded writes or deletes, they need to be synchronized
properly. Also, if you get an error updating one of the indices after
having updated another for the same new doc, you have to recover
properly. If you can manage these issues in your app, then
ParallelReader does exactly what you want.

Chuck


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org