You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Albert Vila <av...@imente.com> on 2005/05/18 16:54:57 UTC
Retrieve all terms
Hi all,
I need to retrieve all terms from an specified field filtered for
another field. For example,
Document 1 -> <contents, " document 1 content">
<language, en>
Document 2 -> <contents, " document 2 content">
<language, fr>
Document 3 -> <contents, " document 3 content">
<language, fr>
Document 4 -> <contents, " document 4 content">
<language, en>
Then, I want to retrieve all terms from the contents field, but only
the ones from the documents matching the language=en.
Is it possible with lucene?
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Retrieve all terms
Posted by Andy Roberts <ma...@andy-roberts.net>.
On Thursday 19 May 2005 06:53, Morus Walter wrote:
> I think he doesn't want the contents but a term list for these contents.
> Something like
> 1 1
> 4 1
> content 2
> document 2
> for his sample, where the number is the fequency of the term.
>
> I don't think that you can easily get that from one lucene index.
> The easiest way to get a term listing for one field of one document is
> to use the term vector support. But for a document collection that would
> still mean to join all term vectors of all matched documents.
I'm not sure if this helps. I have a method that I used when experimenting
with Lucene. It takes in a String which contains the path for an index, opens
a reader, enumerates each term in that index. For each term, it prints out
the term itself its frequency, and the number of documents that term appeared
in.
public static void viewTerms(String indexPath) throws IOException {
IndexReader reader = IndexReader.open(indexPath);
TermEnum te = reader.terms();
while (te.next()) {
Term currentTerm = te.term();
TermPositions tp = reader.termPositions(currentTerm);
int termFreq = 0;
while (tp.next()) {
termFreq += tp.freq();
}
System.out.println(currentTerm.text() + "(" + termFreq + "|" + te.docFreq()
+ ")");
}
reader.close();
}
HTH,
Andy
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Retrieve all terms
Posted by Chris Hostetter <ho...@fucit.org>.
: > > Then, I want to retrieve all terms from the contents field, but
: > > only the ones from the documents matching the language=en.
I think what you want is to...
1) open an IndexReader r.
2) create a QueryFilter q wrapping a TermQuery for language:en
3) BitSet en = q.bits(r);
4) open a TermEnum using r.terms()
5) for each term, get a TermDocs positioned at that term.
5) for each doc in the TermDocs, if the corrisponding bit is set in "en"
then print the term, and advance your TermEnum; if not, then advance
your TermDoc.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Retrieve all terms
Posted by Morus Walter <mo...@tanto.de>.
Bill Tschumy writes:
>
> On May 18, 2005, at 9:54 AM, Albert Vila wrote:
>
> > Hi all,
> >
> > I need to retrieve all terms from an specified field filtered for
> > another field. For example,
> >
> > Document 1 -> <contents, " document 1 content">
> > <language, en>
> >
> > Document 2 -> <contents, " document 2 content">
> > <language, fr>
> >
> > Document 3 -> <contents, " document 3 content">
> > <language, fr>
> >
> > Document 4 -> <contents, " document 4 content">
> > <language, en>
> >
> > Then, I want to retrieve all terms from the contents field, but
> > only the ones from the documents matching the language=en.
> >
> > Is it possible with lucene?
> > Thanks
>
> Unless I'm misunderstanding your request, not only is it possible,
> this is what Lucene is designed for. Just search for all documents
> with language=en and then iterate over the hits extracting the
> contents of the desired field.
>
I think he doesn't want the contents but a term list for these contents.
Something like
1 1
4 1
content 2
document 2
for his sample, where the number is the fequency of the term.
I don't think that you can easily get that from one lucene index.
The easiest way to get a term listing for one field of one document is
to use the term vector support. But for a document collection that would
still mean to join all term vectors of all matched documents.
I would suggest to index the different collections in separated indexes.
Then you can simply loop over all terms.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Retrieve all terms
Posted by Bill Tschumy <bi...@otherwise.com>.
On May 18, 2005, at 9:54 AM, Albert Vila wrote:
> Hi all,
>
> I need to retrieve all terms from an specified field filtered for
> another field. For example,
>
> Document 1 -> <contents, " document 1 content">
> <language, en>
>
> Document 2 -> <contents, " document 2 content">
> <language, fr>
>
> Document 3 -> <contents, " document 3 content">
> <language, fr>
>
> Document 4 -> <contents, " document 4 content">
> <language, en>
>
> Then, I want to retrieve all terms from the contents field, but
> only the ones from the documents matching the language=en.
>
> Is it possible with lucene?
> Thanks
Unless I'm misunderstanding your request, not only is it possible,
this is what Lucene is designed for. Just search for all documents
with language=en and then iterate over the hits extracting the
contents of the desired field.
--
Bill Tschumy
Otherwise -- Austin, TX
http://www.otherwise.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org