You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Venkateshprasanna <pr...@yahoo.co.in> on 2006/12/13 12:02:46 UTC
Extracting data from Lucene index files
I would like to use the data stored in the Lucene indexes, like the words and
their frequencies and store them in a database. Can anyone suggest a way of
going about it or is it possible at all?
TIA
Prasanna
--
View this message in context: http://www.nabble.com/Extracting-data-from-Lucene-index-files-tf2813318.html#a7850919
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Extracting data from Lucene index files
Posted by Venkateshprasanna <pr...@yahoo.co.in>.
Thanks a lot Doron, it worked fine and thanks for your tip as well!
Prasanna
Using term vectors means passing on the terms too many times - i.e
- loop on terms
- - loop on docs of a term
- - - loop on terms of a doc
Would something like this be better:
do {
System.out.println(tenum.term()+" appears in "+tenum.docFreq()+"
docs!");
TermDocs td = reader.termDocs(tenum.term());
do {
System.out.println(" In doc id: "+td.doc() + " it appears: " +
td.freq()+ " times");
} while (td.next());
} while (tenum.next());
Also, you can skip faster to a certain doc (id) or certain term using the
skipTo() methods.
Doron
--
View this message in context: http://www.nabble.com/Extracting-data-from-Lucene-index-files-tf2813318.html#a8050926
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Extracting data from Lucene index files
Posted by Doron Cohen <DO...@il.ibm.com>.
Using term vectors means passing on the terms too many times - i.e
- loop on terms
- - loop on docs of a term
- - - loop on terms of a doc
Would something like this be better:
do {
System.out.println(tenum.term()+" appears in "+tenum.docFreq()+"
docs!");
TermDocs td = reader.termDocs(tenum.term());
do {
System.out.println(" In doc id: "+td.doc() + " it appears: " +
td.freq()+ " times");
} while (td.next());
} while (tenum.next());
Also, you can skip faster to a certain doc (id) or certain term using the
skipTo() methods.
Doron
Venkateshprasanna <pr...@yahoo.co.in> wrote on 19/12/2006 19:20:52:
>
> > Take a look at TermDocs and TermEnum.
>
> I need to get the frequency of each word in each of the documents I have
> indexed.
>
> This is what I could do with TermEnums and TermDocs. For each Term from
> TermEnum, I have instantiated a TermsDoc and for each doc, I am trying to
> get the frequency of the Term.
>
> IndexReader ir = IndexReader.open("index file");
> TermEnum terms = ir.terms();
> while(terms.next()) {
> TermDocs docs = ir.termDocs(terms.term());
>
> while(docs.next()) {
> TermFreqVector tfv =
ir.getTermFreqVector(docs.doc(),"contents");
> String indexTerms[] = tfv.getTerms();
> int indexFreqs[] = tfv.getTermFrequencies();
>
> for(int i = 0; i<indexTerms.length; i++) {
> System.out.println(indexTerms[i]+" "+indexFreqs[i]);
> }
> }
> }
>
> But there is no way of getting the frequency of only 'that' term in
'that'
> document. I have to get the entire vector. This puts the loop in
jeopardy.
> How can I overcome this?
>
> --
> View this message in context: http://www.nabble.com/Extracting-data-
> from-Lucene-index-files-tf2813318.html#a7984092
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Extracting data from Lucene index files
Posted by Venkateshprasanna <pr...@yahoo.co.in>.
> Take a look at TermDocs and TermEnum.
I need to get the frequency of each word in each of the documents I have
indexed.
This is what I could do with TermEnums and TermDocs. For each Term from
TermEnum, I have instantiated a TermsDoc and for each doc, I am trying to
get the frequency of the Term.
IndexReader ir = IndexReader.open("index file");
TermEnum terms = ir.terms();
while(terms.next()) {
TermDocs docs = ir.termDocs(terms.term());
while(docs.next()) {
TermFreqVector tfv = ir.getTermFreqVector(docs.doc(),"contents");
String indexTerms[] = tfv.getTerms();
int indexFreqs[] = tfv.getTermFrequencies();
for(int i = 0; i<indexTerms.length; i++) {
System.out.println(indexTerms[i]+" "+indexFreqs[i]);
}
}
}
But there is no way of getting the frequency of only 'that' term in 'that'
document. I have to get the entire vector. This puts the loop in jeopardy.
How can I overcome this?
--
View this message in context: http://www.nabble.com/Extracting-data-from-Lucene-index-files-tf2813318.html#a7984092
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Extracting data from Lucene index files
Posted by Grant Ingersoll <gs...@apache.org>.
Take a look at TermDocs and TermEnum.
-Grant
On Dec 13, 2006, at 6:02 AM, Venkateshprasanna wrote:
>
> I would like to use the data stored in the Lucene indexes, like the
> words and
> their frequencies and store them in a database. Can anyone suggest
> a way of
> going about it or is it possible at all?
>
> TIA
> Prasanna
> --
> View this message in context: http://www.nabble.com/Extracting-data-
> from-Lucene-index-files-tf2813318.html#a7850919
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org