You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Matt Magoffin <ap...@msqr.us> on 2005/11/10 00:48:14 UTC
efficiently finding all terms used on a particular field within
Documents matching a query
I've used the example posted at
http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-a801793d7479264e29157d92440199b35266dc18
to find all terms used in a complete index, but was wondering if there is
an efficient way to find all terms used within only a set of Documents
matching a query?
For example I would like to find the set of terms used within a particular
date range, where all Documents have a date field on them. What I've done
to date is simply perform a query to find all Documents that match the
date range query, then iterate over each one and construct a Set of all
terms used in the particular field I'm interested in.
I'm wondering if there a more efficient way to accomplish this?
-- m@
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: efficiently finding all terms used on a particular field within Documents matching a query
Posted by Paul Elschot <pa...@xs4all.nl>.
On Thursday 10 November 2005 08:12, Chris Hostetter wrote:
>
> : For example I would like to find the set of terms used within a particular
> : date range, where all Documents have a date field on them. What I've done
> : to date is simply perform a query to find all Documents that match the
> : date range query, then iterate over each one and construct a Set of all
> : terms used in the particular field I'm interested in.
> :
> : I'm wondering if there a more efficient way to accomplish this?
>
> I believe there is -- provided the terms are index.
>
> 1) Get yourself a BitSet representing the Documents you are interested in
> (you mentioned having a a date range, you can either use a RangeFilter nad
> call the bits method directly, or you can do a search using a
> HitCollector)
>
> 2) Look at the code that acctually makes RangeFilter work. It iterates
> over a TermEnum between a low and high value. for each term it finds, it
> uses a TermDocs to record the docid. You could do something very similar,
> looping over all terms in the field you want. but instead of recording
> the docid, add the term to your Set object -- if and only if one of the
> docids from the TermDocs is in your BitSet from step #1 above.
Once the doc id's are available, an alternative to get the terms used
in a particular field is to get them from the TermVectors of the docs.
The field of interest must be indexed with TermVectors for this to work.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: efficiently finding all terms used on a particular field
withinDocuments matching a query
Posted by Chris Hostetter <ho...@fucit.org>.
: Thank you, I had thought a BitSet was appropriate here somehow, I'll work
: on this approach.
Paul's suggestion is acctually a lot simpler, and I suspect it might be
faster -- but it does require that you index with TermVectors. If that's
soemthing you're already doing, then you should definitely try that
appraoch first, if not ... i don't know which one would be best. adding
TermVectors would definitely increase the size of your index, which would
probably affect the performance of your application in other ways.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: efficiently finding all terms used on a particular field
withinDocuments matching a query
Posted by Matt Magoffin <ap...@msqr.us>.
> : I'm wondering if there a more efficient way to accomplish this?
>
> I believe there is -- provided the terms are index.
>
> 1) Get yourself a BitSet representing the Documents you are interested in
> (you mentioned having a a date range, you can either use a RangeFilter nad
> call the bits method directly, or you can do a search using a
> HitCollector)
>
> 2) Look at the code that acctually makes RangeFilter work. It iterates
> over a TermEnum between a low and high value. for each term it finds, it
> uses a TermDocs to record the docid. You could do something very similar,
> looping over all terms in the field you want. but instead of recording
> the docid, add the term to your Set object -- if and only if one of the
> docids from the TermDocs is in your BitSet from step #1 above.
>
>
> ...that should be faster then the appraoch you have now.
Thank you, I had thought a BitSet was appropriate here somehow, I'll work
on this approach.
-- m@
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: efficiently finding all terms used on a particular field within
Documents matching a query
Posted by Chris Hostetter <ho...@fucit.org>.
: For example I would like to find the set of terms used within a particular
: date range, where all Documents have a date field on them. What I've done
: to date is simply perform a query to find all Documents that match the
: date range query, then iterate over each one and construct a Set of all
: terms used in the particular field I'm interested in.
:
: I'm wondering if there a more efficient way to accomplish this?
I believe there is -- provided the terms are index.
1) Get yourself a BitSet representing the Documents you are interested in
(you mentioned having a a date range, you can either use a RangeFilter nad
call the bits method directly, or you can do a search using a
HitCollector)
2) Look at the code that acctually makes RangeFilter work. It iterates
over a TermEnum between a low and high value. for each term it finds, it
uses a TermDocs to record the docid. You could do something very similar,
looping over all terms in the field you want. but instead of recording
the docid, add the term to your Set object -- if and only if one of the
docids from the TermDocs is in your BitSet from step #1 above.
...that should be faster then the appraoch you have now.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org