You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bob Daha <ne...@yahoo.com> on 2007/12/10 20:56:46 UTC
Out of memory?
Hello,
I'm building a ticketing system for my company and am using Lucene for some of the more complicated queries. I'd say my application differs from the typical lucene application in that my documents are (re)-indexed more frequently, the query load is actually relatively light, and most of the indexed document data is numerical. (ticket id, owner id, date ranges, etc are numbers and there's just one giant "text" searchable field.)
I just got an out of memory exception on my app. When I fired up jhat, I was very surprised at the object allocation:
7862 instances of class org.apache.lucene.index.Term
7787 instances of class org.apache.lucene.index.TermInfo
504 instances of class org.apache.lucene.index.FieldInfo
224 instances of class org.apache.lucene.index.TermBuffer
79 instances of class org.apache.lucene.index.CompoundFileReader$CSIndexInput
75 instances of class org.apache.lucene.index.SegmentTermEnum
70 instances of class org.apache.lucene.store.RAMFile
64 instances of class org.apache.lucene.store.RAMInputStream
63 instances of class org.apache.lucene.index.FieldInfos
20 instances of class com.facebook.tps_search_server.tps_search_ticket
20 instances of class com.facebook.tps_search_server.tps_search_ticket$Isset
10 instances of class org.apache.lucene.index.IndexReader$FieldOption
9 instances of class org.apache.lucene.index.IndexFileDeleter$RefCount
8 instances of class org.apache.lucene.index.CompoundFileReader$FileEntry
8 instances of class org.apache.lucene.index.CompoundFileWriter$FileEntry
8 instances of class org.apache.lucene.index.SegmentReader$Norm
7 instances of class org.apache.lucene.document.FieldSelectorResult
5 instances of class org.apache.lucene.document.Field$TermVector
5 instances of class org.apache.lucene.queryParser.Token
4 instances of class org.apache.lucene.document.Field$Index
4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry
3 instances of class com.facebook.thrift.protocol.TBinaryProtocol
3 instances of class com.facebook.thrift.transport.TSocket
3 instances of class org.apache.lucene.document.Field$Store
3 instances of class org.apache.lucene.index.SegmentInfos
3 instances of class org.apache.lucene.search.BooleanClause$Occur
2 instances of class com.facebook.thrift.protocol.TBinaryProtocol$Factory
2 instances of class com.facebook.thrift.server.TThreadPoolServer$WorkerProcess
2 instances of class com.facebook.thrift.transport.TTransportFactory
2 instances of class org.apache.lucene.analysis.standard.StandardAnalyzer
2 instances of class org.apache.lucene.index.SegmentInfo
2 instances of class org.apache.lucene.queryParser.QueryParser$Operator
2 instances of class org.apache.lucene.search.DefaultSimilarity
2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2
2 instances of class org.apache.lucene.search.Sort
2 instances of class org.apache.lucene.search.SortField
2 instances of class org.apache.lucene.store.RAMDirectory
2 instances of class org.apache.lucene.store.SingleInstanceLockFactory
2 instances of class [Lorg.apache.lucene.search.SortField;
...
Any ideas on why I'd have so many Term and TermInfo objects? To give you a little more insight into my application, worker threads can be spawned by client requests that do one of 4 things - search, index tickets, deindex tickets, or reindex tickets. (reindex a ticket is basically deindex then index ticket) I'm using a RAMDirectory for all my searches and updates although I have another thread which backs this up to disc periodically.
Thanks,
B
____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
Re: Out of memory?
Posted by Grant Ingersoll <gs...@apache.org>.
Those terms could be coming from a lot of diff. places.
What size JVM heap, etc. are you giving your application? My guess is
your index is too big to fit into RAM, you probably need to use a
FSDirectory or give it more RAM, but it is hard to say for sure w/o
knowing more about the application.
See http://wiki.apache.org/lucene-java/BasicsOfPerformance for more on
performance.
-Grant
On Dec 10, 2007, at 2:56 PM, Bob Daha wrote:
> Hello,
>
> I'm building a ticketing system for my company and am using Lucene
> for some of the more complicated queries. I'd say my application
> differs from the typical lucene application in that my documents are
> (re)-indexed more frequently, the query load is actually relatively
> light, and most of the indexed document data is numerical. (ticket
> id, owner id, date ranges, etc are numbers and there's just one
> giant "text" searchable field.)
>
> I just got an out of memory exception on my app. When I fired up
> jhat, I was very surprised at the object allocation:
>
> 7862 instances of class org.apache.lucene.index.Term
> 7787 instances of class org.apache.lucene.index.TermInfo
> 504 instances of class org.apache.lucene.index.FieldInfo
> 224 instances of class org.apache.lucene.index.TermBuffer
> 79 instances of class org.apache.lucene.index.CompoundFileReader
> $CSIndexInput
> 75 instances of class org.apache.lucene.index.SegmentTermEnum
> 70 instances of class org.apache.lucene.store.RAMFile
> 64 instances of class org.apache.lucene.store.RAMInputStream
> 63 instances of class org.apache.lucene.index.FieldInfos
> 20 instances of class com.facebook.tps_search_server.tps_search_ticket
> 20 instances of class
> com.facebook.tps_search_server.tps_search_ticket$Isset
> 10 instances of class org.apache.lucene.index.IndexReader$FieldOption
> 9 instances of class org.apache.lucene.index.IndexFileDeleter$RefCount
> 8 instances of class org.apache.lucene.index.CompoundFileReader
> $FileEntry
> 8 instances of class org.apache.lucene.index.CompoundFileWriter
> $FileEntry
> 8 instances of class org.apache.lucene.index.SegmentReader$Norm
> 7 instances of class org.apache.lucene.document.FieldSelectorResult
> 5 instances of class org.apache.lucene.document.Field$TermVector
> 5 instances of class org.apache.lucene.queryParser.Token
> 4 instances of class org.apache.lucene.document.Field$Index
> 4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry
> 3 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> 3 instances of class com.facebook.thrift.transport.TSocket
> 3 instances of class org.apache.lucene.document.Field$Store
> 3 instances of class org.apache.lucene.index.SegmentInfos
> 3 instances of class org.apache.lucene.search.BooleanClause$Occur
> 2 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> $Factory
> 2 instances of class com.facebook.thrift.server.TThreadPoolServer
> $WorkerProcess
> 2 instances of class com.facebook.thrift.transport.TTransportFactory
> 2 instances of class
> org.apache.lucene.analysis.standard.StandardAnalyzer
> 2 instances of class org.apache.lucene.index.SegmentInfo
> 2 instances of class org.apache.lucene.queryParser.QueryParser
> $Operator
> 2 instances of class org.apache.lucene.search.DefaultSimilarity
> 2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2
> 2 instances of class org.apache.lucene.search.Sort
> 2 instances of class org.apache.lucene.search.SortField
> 2 instances of class org.apache.lucene.store.RAMDirectory
> 2 instances of class org.apache.lucene.store.SingleInstanceLockFactory
> 2 instances of class [Lorg.apache.lucene.search.SortField;
> ...
>
> Any ideas on why I'd have so many Term and TermInfo objects? To
> give you a little more insight into my application, worker threads
> can be spawned by client requests that do one of 4 things - search,
> index tickets, deindex tickets, or reindex tickets. (reindex a
> ticket is basically deindex then index ticket) I'm using a
> RAMDirectory for all my searches and updates although I have another
> thread which backs this up to disc periodically.
>
> Thanks,
> B
>
>
>
>
>
> ____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Out of memory?
Posted by Chris Lu <ch...@gmail.com>.
Looks like you are using FieldCacheImpl to count search results for
each category. ( or called facet search by a fancy name ).
Well, it's a cache and the terms are loaded in the memory.
--
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer Twenga got 2.6 Million Euro funding!
http://www.techcrunch.com/2007/12/07/twenga-gets-e26-million-for-product-search/
On Dec 10, 2007 11:56 AM, Bob Daha <ne...@yahoo.com> wrote:
> Hello,
>
> I'm building a ticketing system for my company and am using Lucene for some of the more complicated queries. I'd say my application differs from the typical lucene application in that my documents are (re)-indexed more frequently, the query load is actually relatively light, and most of the indexed document data is numerical. (ticket id, owner id, date ranges, etc are numbers and there's just one giant "text" searchable field.)
>
> I just got an out of memory exception on my app. When I fired up jhat, I was very surprised at the object allocation:
>
> 7862 instances of class org.apache.lucene.index.Term
> 7787 instances of class org.apache.lucene.index.TermInfo
> 504 instances of class org.apache.lucene.index.FieldInfo
> 224 instances of class org.apache.lucene.index.TermBuffer
> 79 instances of class org.apache.lucene.index.CompoundFileReader$CSIndexInput
> 75 instances of class org.apache.lucene.index.SegmentTermEnum
> 70 instances of class org.apache.lucene.store.RAMFile
> 64 instances of class org.apache.lucene.store.RAMInputStream
> 63 instances of class org.apache.lucene.index.FieldInfos
> 20 instances of class com.facebook.tps_search_server.tps_search_ticket
> 20 instances of class com.facebook.tps_search_server.tps_search_ticket$Isset
> 10 instances of class org.apache.lucene.index.IndexReader$FieldOption
> 9 instances of class org.apache.lucene.index.IndexFileDeleter$RefCount
> 8 instances of class org.apache.lucene.index.CompoundFileReader$FileEntry
> 8 instances of class org.apache.lucene.index.CompoundFileWriter$FileEntry
> 8 instances of class org.apache.lucene.index.SegmentReader$Norm
> 7 instances of class org.apache.lucene.document.FieldSelectorResult
> 5 instances of class org.apache.lucene.document.Field$TermVector
> 5 instances of class org.apache.lucene.queryParser.Token
> 4 instances of class org.apache.lucene.document.Field$Index
> 4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry
> 3 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> 3 instances of class com.facebook.thrift.transport.TSocket
> 3 instances of class org.apache.lucene.document.Field$Store
> 3 instances of class org.apache.lucene.index.SegmentInfos
> 3 instances of class org.apache.lucene.search.BooleanClause$Occur
> 2 instances of class com.facebook.thrift.protocol.TBinaryProtocol$Factory
> 2 instances of class com.facebook.thrift.server.TThreadPoolServer$WorkerProcess
> 2 instances of class com.facebook.thrift.transport.TTransportFactory
> 2 instances of class org.apache.lucene.analysis.standard.StandardAnalyzer
> 2 instances of class org.apache.lucene.index.SegmentInfo
> 2 instances of class org.apache.lucene.queryParser.QueryParser$Operator
> 2 instances of class org.apache.lucene.search.DefaultSimilarity
> 2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2
> 2 instances of class org.apache.lucene.search.Sort
> 2 instances of class org.apache.lucene.search.SortField
> 2 instances of class org.apache.lucene.store.RAMDirectory
> 2 instances of class org.apache.lucene.store.SingleInstanceLockFactory
> 2 instances of class [Lorg.apache.lucene.search.SortField;
> ...
>
> Any ideas on why I'd have so many Term and TermInfo objects? To give you a little more insight into my application, worker threads can be spawned by client requests that do one of 4 things - search, index tickets, deindex tickets, or reindex tickets. (reindex a ticket is basically deindex then index ticket) I'm using a RAMDirectory for all my searches and updates although I have another thread which backs this up to disc periodically.
>
> Thanks,
> B
>
>
>
>
> ____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org