You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Chan <mc...@cse.unsw.edu.au> on 2006/05/01 21:37:50 UTC

Stemming terms in SpanQuery

Hi,

I'm trying to build a SpanQuery using word stems. Is parsing each term 
with a QueryParser, constructed with an Analyzer giving stemmed 
tokenStream, the right approach? It just seems to me that QueryParser is 
designed to parse queries, and so my hunch is that there might be a 
better way.

Any help will be much appreciated.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Updating index if there is a database changes

Posted by Kiran Joisher <ki...@recreatesolutions.com>.
Thanks Stephen,

This was really helpful. 

Cheers,
--Kiran


 
-----Original Message-----
From: Stephen Gray [mailto:Stephen.Gray@anu.edu.au] 
Sent: Thursday, May 04, 2006 4:11 AM
To: java-user@lucene.apache.org
Subject: Re: Updating index if there is a database changes

Hi Kirin,

Once you've updated an index using IndexWriter or IndexReader you just need
to close and re-open your IndexSearcher so that searching includes the
changes. There is a great library callled LuceIndexAccessor at the link
below that manages this for you. It creates an IndexReader/Writer/Searcher
and hands out references to classes that need to use them. When you want to
read/write you just get a reference to the appropriate object from the
index accessor, then release it when you've finished. If you make changes
using an IndexWriter then release the reference, it waits until all
IndexSearcher refs have been released, then recreates the searcher
automatically.

The link is:
http://www.nabble.com/Fwd%3A-Contribution%3A-LuceneIndexAccessor-t17416.html
#a47049

You might also like to look at DelayCloseIndexSearcher at
http://issues.apache.org/jira/browse/LUCENE-445

Regards,
Steve

At 05:31 PM 3/05/2006, you wrote:
>Hi all,
>
>I m working on a project where I will use lucene to make a search engine on
>a database. I am new to lucene. I wrote a test program which indexes a
table
>and searches the same.. but now I m stuck on how to update the index in
case
>a database change occurs.. I need some help on this topic...like how do I
>update the index at run time... can it be done then and there...or do I
have
>to write some kind of schedular program which "re-builds" the entire index
>say once in a day ... which will be more efficient ?
>
>the data will be huge... 4 million records something..
>
>Thanks in advance,
>--Kiran
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org

Stephen Gray
Archive Research Officer
Australian Social Science Data Archive
18 Balmain Crescent (Building #66)
The Australian National University
Canberra ACT 0200

Phone +61 2 6125 2185
Fax +61 2 6125 0627
Web http://assda.anu.edu.au/

**************************************************************
Scanned by  eScan  Anti-Virus  and  Content Security Software.
Visit http://www.mwti.net for more info on eScan and MailScan.
**************************************************************



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Updating index if there is a database changes

Posted by Stephen Gray <St...@anu.edu.au>.
Hi Kirin,

Once you've updated an index using IndexWriter or IndexReader you just need 
to close and re-open your IndexSearcher so that searching includes the 
changes. There is a great library callled LuceIndexAccessor at the link 
below that manages this for you. It creates an IndexReader/Writer/Searcher 
and hands out references to classes that need to use them. When you want to 
read/write you just get a reference to the appropriate object from the 
index accessor, then release it when you've finished. If you make changes 
using an IndexWriter then release the reference, it waits until all 
IndexSearcher refs have been released, then recreates the searcher 
automatically.

The link is:
http://www.nabble.com/Fwd%3A-Contribution%3A-LuceneIndexAccessor-t17416.html#a47049

You might also like to look at DelayCloseIndexSearcher at 
http://issues.apache.org/jira/browse/LUCENE-445

Regards,
Steve

At 05:31 PM 3/05/2006, you wrote:
>Hi all,
>
>I m working on a project where I will use lucene to make a search engine on
>a database. I am new to lucene. I wrote a test program which indexes a table
>and searches the same.. but now I m stuck on how to update the index in case
>a database change occurs.. I need some help on this topic...like how do I
>update the index at run time... can it be done then and there...or do I have
>to write some kind of schedular program which "re-builds" the entire index
>say once in a day ... which will be more efficient ?
>
>the data will be huge... 4 million records something..
>
>Thanks in advance,
>--Kiran
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org

Stephen Gray
Archive Research Officer
Australian Social Science Data Archive
18 Balmain Crescent (Building #66)
The Australian National University
Canberra ACT 0200

Phone +61 2 6125 2185
Fax +61 2 6125 0627
Web http://assda.anu.edu.au/

Re: Updating index if there is a database changes

Posted by Yonik Seeley <ys...@gmail.com>.
On 5/3/06, Kiran Joisher <ki...@recreatesolutions.com> wrote:
> I m working on a project where I will use lucene to make a search engine on
> a database. I am new to lucene. I wrote a test program which indexes a table
> and searches the same.. but now I m stuck on how to update the index in case
> a database change occurs.. I need some help on this topic...

Solr handles incremental updating:
  - overwriting documents that already exist in the collection
  - graceful migration from old IndexSearchers to new IndexSearchers
so that the changes are visible

Solr does not yet handle automatically grabbing stuff from a
database... you need to write something that grabs the changes and
then sends those changes to Solr (via XML over HTTP-POST)


-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Updating index if there is a database changes

Posted by Chris Lu <ch...@gmail.com>.
My approach is to select documents ordered by updated_date desc
And only process documents newer than the ones already in the index.

Chris Lu
------------------------------------
Full-Text Lucene Search for Any Databases/Applications
http://www.dbsight.net


On 5/3/06, Kiran Joisher <ki...@recreatesolutions.com> wrote:
> Hi all,
>
> I m working on a project where I will use lucene to make a search engine on
> a database. I am new to lucene. I wrote a test program which indexes a table
> and searches the same.. but now I m stuck on how to update the index in case
> a database change occurs.. I need some help on this topic...like how do I
> update the index at run time... can it be done then and there...or do I have
> to write some kind of schedular program which "re-builds" the entire index
> say once in a day ... which will be more efficient ?
>
> the data will be huge... 4 million records something..
>
> Thanks in advance,
> --Kiran
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Updating index if there is a database changes

Posted by Kiran Joisher <ki...@recreatesolutions.com>.
Hi all,

I m working on a project where I will use lucene to make a search engine on
a database. I am new to lucene. I wrote a test program which indexes a table
and searches the same.. but now I m stuck on how to update the index in case
a database change occurs.. I need some help on this topic...like how do I
update the index at run time... can it be done then and there...or do I have
to write some kind of schedular program which "re-builds" the entire index
say once in a day ... which will be more efficient ?

the data will be huge... 4 million records something.. 

Thanks in advance,
--Kiran




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Stemming terms in SpanQuery

Posted by Jason Calabrese <ma...@jasoncalabrese.com>.
I think the best way to tokening/stem is to use the analyzer directly.  for 
example:

TokenStream ts = analyzer.tokenStream(field, new StringReader(text));
				
Token token = null;
while ((token = ts.next()) != null) {
	Term newTerm = new Term(field, token.termText());
	//do something with the new newTerm
}

If your just stemming then there will be only a single token so you probably 
won't need the while loop.

On Monday 01 May 2006 12:37, Michael Chan wrote:
> Hi,
>
> I'm trying to build a SpanQuery using word stems. Is parsing each term
> with a QueryParser, constructed with an Analyzer giving stemmed
> tokenStream, the right approach? It just seems to me that QueryParser is
> designed to parse queries, and so my hunch is that there might be a
> better way.
>
> Any help will be much appreciated.
>
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: creating indexReader object

Posted by trupti mulajkar <ac...@sheffield.ac.uk>.
i have indexed files uisng IndexFiles, 
how can i add the field to the document using this.

cheers,
trupti mulajkar
MSc Advanced Computer Science


Quoting karl wettin <ka...@snigel.net>:

> 
> 2 maj 2006 kl. 16.11 skrev trupti mulajkar:
> >
> > doc(i).get("contents");
> >
> > i get an only NULL
> >
> > any ideas ?
> 
> Did you index the field with term vector when you added it to the  
> document?
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: creating indexReader object

Posted by karl wettin <ka...@snigel.net>.
2 maj 2006 kl. 16.11 skrev trupti mulajkar:
>
> doc(i).get("Contents");
>
> i get an only NULL
>
> any ideas ?

Did you index the field with term vector when you added it to the  
document?


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: creating indexReader object

Posted by Frank Kunemann <fr...@innosystec.de>.
Lucene's fields are case sensitive and I think "contents" is written in
lower case by default.

Cheers,
Frank


-----Original Message-----
From: trupti mulajkar [mailto:acp05tm@sheffield.ac.uk] 
Sent: Tuesday, May 02, 2006 4:11 PM
To: java-user@lucene.apache.org
Subject: Re: creating indexReader object

thanx hannes, 

but i dont think i made my query clear enough.
i have created the index reader object just the way you mentioned it, but
after that when i try to do create the vectors like term frequency and
document frequency using

doc(i).get("Contents"); 

i get an only NULL

any ideas ?

cheers,
trupti mulajkar
MSc Advanced Computer Science


Quoting Hannes Carl Meyer <de...@rc.ag>:

> Hi,
> 
> IndexReader has some static methods, e.g.
> 
> IndexReader reader = IndexReader.open(new File("/index"));
>
http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexReader.h
tml#open(java.lang.String)
> 
> Hannes
> 
> trupti mulajkar schrieb:
> > i am trying to create an object of index reader class that reads my
index.
> i
> > need this to further generate the document and term frequency vectors.
> > however when i try to print the contents of the documents
> (doc.get("contents"))
> > it shows -null .
> > any suggestions,
> > if i cant read the contents then i cannot create the other vectors.
> >
> > any help will be apprecisted
> >
> > cheers,
> > trupti mulajkar
> > MSc Advanced Computer Science
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >   
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: creating indexReader object

Posted by trupti mulajkar <ac...@sheffield.ac.uk>.
thanx hannes, 

but i dont think i made my query clear enough.
i have created the index reader object just the way you mentioned it, but after 
that when i try to do create the vectors like term frequency and document
frequency using

doc(i).get("Contents"); 

i get an only NULL

any ideas ?

cheers,
trupti mulajkar
MSc Advanced Computer Science


Quoting Hannes Carl Meyer <de...@rc.ag>:

> Hi,
> 
> IndexReader has some static methods, e.g.
> 
> IndexReader reader = IndexReader.open(new File("/index"));
>
http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexReader.html#open(java.lang.String)
> 
> Hannes
> 
> trupti mulajkar schrieb:
> > i am trying to create an object of index reader class that reads my index.
> i
> > need this to further generate the document and term frequency vectors.
> > however when i try to print the contents of the documents
> (doc.get("contents"))
> > it shows -null .
> > any suggestions, 
> > if i cant read the contents then i cannot create the other vectors.
> >
> > any help will be apprecisted
> >
> > cheers,
> > trupti mulajkar
> > MSc Advanced Computer Science
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >   
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: creating indexReader object

Posted by Hannes Carl Meyer <de...@rc.ag>.
Hi,

IndexReader has some static methods, e.g.

IndexReader reader = IndexReader.open(new File("/index"));
http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexReader.html#open(java.lang.String)

Hannes

trupti mulajkar schrieb:
> i am trying to create an object of index reader class that reads my index. i
> need this to further generate the document and term frequency vectors.
> however when i try to print the contents of the documents (doc.get("contents"))
> it shows -null .
> any suggestions, 
> if i cant read the contents then i cannot create the other vectors.
>
> any help will be apprecisted
>
> cheers,
> trupti mulajkar
> MSc Advanced Computer Science
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


creating indexReader object

Posted by trupti mulajkar <ac...@sheffield.ac.uk>.
i am trying to create an object of index reader class that reads my index. i
need this to further generate the document and term frequency vectors.
however when i try to print the contents of the documents (doc.get("contents"))
it shows -null .
any suggestions, 
if i cant read the contents then i cannot create the other vectors.

any help will be apprecisted

cheers,
trupti mulajkar
MSc Advanced Computer Science



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by Ramana Jelda <ra...@ciao-group.com>.
Thx  for ur quick reply.
I will go through it.

Rgds,
Jelda 

> -----Original Message-----
> From: mark harwood [mailto:markharw00d@yahoo.co.uk] 
> Sent: Tuesday, May 02, 2006 5:03 PM
> To: java-user@lucene.apache.org
> Subject: RE: OutOfMemoryError while enumerating through 
> reader.terms(fieldName)
> 
> "Category counts" should really be a FAQ entry.
> 
> There is no one right solution to prescribe because it 
> depends on the shape of your data.
> 
> 
> For previous discussions/code samples see here:
> http://www.mail-archive.com/java-user@lucene.apache.org/msg05123.html
> 
> and here for more space-efficient representations for sparse 
> sets 
> http://www.mail-archive.com/java-user@lucene.apache.org/msg02929.html
> 
> 
> Cheers
> Mark
> 
> 
> 		
> ___________________________________________________________
> Switch an email account to Yahoo! Mail, you could win FIFA 
> World Cup tickets. http://uk.mail.yahoo.com
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by mark harwood <ma...@yahoo.co.uk>.
"Category counts" should really be a FAQ entry.

There is no one right solution to prescribe because it
depends on the shape of your data.


For previous discussions/code samples see here:
http://www.mail-archive.com/java-user@lucene.apache.org/msg05123.html

and here for more space-efficient representations for
sparse sets
http://www.mail-archive.com/java-user@lucene.apache.org/msg02929.html


Cheers
Mark


		
___________________________________________________________ 
Switch an email account to Yahoo! Mail, you could win FIFA World Cup tickets. http://uk.mail.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by Ramana Jelda <ra...@ciao-group.com>.
I just got an idea for category counting instead following this BitSet
approach..


I will maintain and array with  docIds to cateogy_ids as value.

i.e. documents[docId] =category_id

Which is taking for 1 million docs,around each docid=4
bytes,category_id=4bytes = 8MBytes 

And then from user query, using HitCollector docids, I will try to calculate
each category count.. I think it is self understandable.. 

What do u think??
Any advice is really welcome..

Note: Actually, I have around 20000 unique cateogry ids.. 

Thx,
Jelda

> -----Original Message-----
> From: Ramana Jelda [mailto:ramana.jelda@ciao-group.com] 
> Sent: Tuesday, May 02, 2006 4:41 PM
> To: java-user@lucene.apache.org
> Subject: RE: OutOfMemoryError while enumerating through 
> reader.terms(fieldName)
> 
> I am trying to implement category count almost similar to 
> CNET approach.
> At the initialization time , I am trying to create all these 
> BitSets and then trying to and them with user query(with a 
> bitset obtained from queryfilter containing user query)..
> 
> This way my application is performant..Don't u think so? 
> Actually I need all those bitsets everytime user queries. I 
> can not use exisiting Lucene filter approach.. Is n't it??
> 
> Thx in advance,
> Jelda
> 
> 
> 
> 
> > -----Original Message-----
> > From: mark harwood [mailto:markharw00d@yahoo.co.uk]
> > Sent: Tuesday, May 02, 2006 4:19 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: OutOfMemoryError while enumerating through
> > reader.terms(fieldName)
> > 
> > >>Any advise is relly welcome.
> > 
> > Don't cache all that data.
> > You need a minimum of (numUniqueTerms*numDocs)/8 bytes to hold that 
> > info.
> > Assuming 10,000 unique terms and 1 million docs you'd need 
> over 1 Gig 
> > of RAM.
> > 
> > I suppose the question is what are you trying to achieve 
> and why can't 
> > you use the existing Lucene APIs instead of caching all 
> those bitsets?
> > 
> > Cheers
> > Mark
> > 
> > 
> > 		
> > ___________________________________________________________
> > Switch an email account to Yahoo! Mail, you could win FIFA 
> World Cup 
> > tickets. http://uk.mail.yahoo.com
> > 
> > 
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by Ramana Jelda <ra...@ciao-group.com>.
I am trying to implement category count almost similar to CNET approach.
At the initialization time , I am trying to create all these BitSets and
then trying to and them with user query(with a bitset obtained from
queryfilter containing user query)..

This way my application is performant..Don't u think so? 
Actually I need all those bitsets everytime user queries. I can not use
exisiting Lucene filter approach.. Is n't it??

Thx in advance,
Jelda




> -----Original Message-----
> From: mark harwood [mailto:markharw00d@yahoo.co.uk] 
> Sent: Tuesday, May 02, 2006 4:19 PM
> To: java-user@lucene.apache.org
> Subject: RE: OutOfMemoryError while enumerating through 
> reader.terms(fieldName)
> 
> >>Any advise is relly welcome.
> 
> Don't cache all that data.
> You need a minimum of (numUniqueTerms*numDocs)/8 bytes to 
> hold that info.
> Assuming 10,000 unique terms and 1 million docs you'd need 
> over 1 Gig of RAM.
> 
> I suppose the question is what are you trying to achieve and 
> why can't you use the existing Lucene APIs instead of caching 
> all those bitsets?
> 
> Cheers
> Mark
> 
> 
> 		
> ___________________________________________________________
> Switch an email account to Yahoo! Mail, you could win FIFA 
> World Cup tickets. http://uk.mail.yahoo.com
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by mark harwood <ma...@yahoo.co.uk>.
>>Any advise is relly welcome.

Don't cache all that data.
You need a minimum of (numUniqueTerms*numDocs)/8 bytes
to hold that info.
Assuming 10,000 unique terms and 1 million docs you'd
need over 1 Gig of RAM.

I suppose the question is what are you trying to
achieve and why can't you use the existing Lucene APIs
instead of caching all those bitsets?

Cheers
Mark


		
___________________________________________________________ 
Switch an email account to Yahoo! Mail, you could win FIFA World Cup tickets. http://uk.mail.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by Ramana Jelda <ra...@ciao-group.com>.
 
Hi,
I just debugged it closely.. Sorry I am getting OutOfMemoryError not because
of reader.terms()
But because of invoking QueryFilter.bits() method for each unique term.
I will try explain u with psuedo code.

 while(term != null){
       if(term.field().equals(name)){
          String termText = term.text();
          keys.addElement(termText);
       }else{
         break;
       }
      if(te.next()){
        term = te.term();
       }else{
      break;
      }
 }

for(Iterator iter = keys.iterator(); iter.hasNext();){
  String termText = (String) iter.next();
 TermQuery termQuery = new TermQuery(new Term(fieldName, termText));
   QueryFilter filter = new QueryFilter(termQuery);
   final BitSet bits;
   bits = filter.bits(ciaoReader.getIndexReader());
   BitSet pr = cache.put(termText, bits);
}
}

Second for loop which gets BitSet using QueryFilter is now throwing
OutOfMemoryError.

Any advise is relly welcome.

Thx,
Jelda



> -----Original Message-----
> From: Ramana Jelda [mailto:ramana.jelda@ciao-group.com] 
> Sent: Tuesday, May 02, 2006 12:55 PM
> To: java-user@lucene.apache.org
> Subject: OutOfMemoryError while enumerating through 
> reader.terms(fieldName)
> 
> Hi,
> I am getting OutOfMemoryError , while enumerating through  
> TermEnum  after invoking reader.terms(fieldName).
> 
> Just to provide you more information, I have almost 10000 
> unique terms in field A. I can successfully enumerate around 
> 5000terms but later I am gettting OutOfMemoryError.
> 
> I set jvm max memory as 512MB , Ofcourse my index is bigger 
> than this memory around 1GB-2GB..
> How can I ask lucene to cleanup loaded index and traverse 
> through remaining terms.. It seems while enumerating memory 
> always grows in steps of some MBs.
> 
> Any help would be really appreciable.
> 
> Thanks in advance,
> Jelda
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


OutOfMemoryError while enumerating through reader.terms(fieldName)

Posted by Ramana Jelda <ra...@ciao-group.com>.
Hi,
I am getting OutOfMemoryError , while enumerating through  TermEnum  after
invoking reader.terms(fieldName).

Just to provide you more information, I have almost 10000 unique terms in
field A. I can successfully enumerate around 5000terms but later I am
gettting OutOfMemoryError.

I set jvm max memory as 512MB , Ofcourse my index is bigger than this memory
around 1GB-2GB..
How can I ask lucene to cleanup loaded index and traverse through remaining
terms.. It seems while enumerating memory always grows in steps of some MBs.

Any help would be really appreciable.

Thanks in advance,
Jelda


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org