You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by sandyg <ga...@gmail.com> on 2008/03/18 14:23:22 UTC
java.lang.OutOfMemoryError: Java heap space when sorting the fields
this is my search content
QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
Query query = parser.parse("1");
Sort sort = new Sort(new SortField(sortField));
Hits hits = searcher.search(query,sort);
And i had huge data about 13 millions of records
i am not sure y its giving outof memory exception and
no exception when no sorting is done
plz some one help me yar
and also if to increase heap space how to increase it programatically i had
command prompt
java -Xms<initial heap size> -Xmx<maximum heap size>
please.....
--
View this message in context: http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16121128.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by Chris Hostetter <ho...@fucit.org>.
: You might think, if I only ask for the top 10 docs, don't i only read 10 field
: values? But of course you don't know what docs will be returned as each search
: comes in...so you have to cache them all.
Arguements have been made in the past that when you have an index
large enough that the FieldCache is prohibitively expensive, and you
expect the number of *matching* documents to be significantly smaller then
the total number of documents, it might make sense to Sort using
LazyLoaded "stored" field values ... it's a pretty radical shift from the
current sorting approach, and would require some explicit work on the part
of the indexing/searching code, but a patch along that lines was created a
while back...
https://issues.apache.org/jira/browse/LUCENE-769
...based on the last comment i posted in that issue, i clearly
thought it showed a lot of promise, but needed cleaned up. I admittedly
lost track of it as other things took priority in life. if people want
to dust it off, try it out, and make comments about it's effectiveness I'm
sure it could be revived.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the fields
Posted by Daniel Noll <da...@nuix.com>.
On Thursday 20 March 2008 07:22:27 Mark Miller wrote:
> You might think, if I only ask for the top 10 docs, don't i only read 10
> field values? But of course you don't know what docs will be returned as
> each search comes in...so you have to cache them all.
If it lazily cached one field at a time, I guess this wouldn't be a problem...
but you would still eventually still run out of memory after performing a
certain number of queries.
I guess one solution would be a FieldCache backed by storage on hard disk.
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by Mark Miller <ma...@gmail.com>.
Heres what happens: in order to sort all of the hits you get back on a
field, you need to get the value of that field for comparisons right?
Well it turns out that reading a field value from the index is pretty
slow (its on the disk after all)...so Lucene will read all of the terms
in the field off disk once (on the first sort/search request) and cache
them. So if your field is an Int your talking numDocs*32 bits for your
cache. For a Long field its numDocs*64. For a String field Lucene caches
a String array with every unique term and then an int array indexing
into the term array.
You might think, if I only ask for the top 10 docs, don't i only read 10
field values? But of course you don't know what docs will be returned as
each search comes in...so you have to cache them all.
Like the other answer said, one field is going to be roughly 50 MB. You
can do the math yourself...I ripped the 40 doing some rough stuff in my
head, but around 50mb is prob closer.
So anyway, you *are* only sorting on the hits you get back...but youll
get different hits all the time so you still have to cache all the field
values. The FieldCache class does this, and thats whats taking up the
RAM. You should have 50 to give it though I would assume...how much RAM
are you giving the JVM?
- mark
sandyg wrote:
> Hi,
> And is it not passibe to sort on the result we get instead of on all the
> values like
> Hits hits = searcher.search(query);
> and it will be good if got sorting on the hits i.e on the result
> because my sorting is based on specific field and that field should be
> sorted when i click on it.
> thanks for the reply
>
>
> markrmiller wrote:
>
>> To sort on 13mil docs will take like at least 400 mb for the field
>> cache. Thats if you only sort on one field...it can grow fast if you
>> allow multi field sorting.
>>
>> How much RAM are you giving your app?
>>
>> sandyg wrote:
>>
>>> this is my search content
>>>
>>> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
>>> Query query = parser.parse("1");
>>>
>>> Sort sort = new Sort(new SortField(sortField));
>>> Hits hits = searcher.search(query,sort);
>>>
>>> And i had huge data about 13 millions of records
>>> i am not sure y its giving outof memory exception and
>>> no exception when no sorting is done
>>> plz some one help me yar
>>>
>>> and also if to increase heap space how to increase it programatically i
>>> had
>>> command prompt
>>> java -Xms<initial heap size> -Xmx<maximum heap size>
>>> please.....
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by sandyg <ga...@gmail.com>.
Hi,
And is it not passibe to sort on the result we get instead of on all the
values like
Hits hits = searcher.search(query);
and it will be good if got sorting on the hits i.e on the result
thanks for the reply
markrmiller wrote:
>
> To sort on 13mil docs will take like at least 400 mb for the field
> cache. Thats if you only sort on one field...it can grow fast if you
> allow multi field sorting.
>
> How much RAM are you giving your app?
>
> sandyg wrote:
>> this is my search content
>>
>> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
>> Query query = parser.parse("1");
>>
>> Sort sort = new Sort(new SortField(sortField));
>> Hits hits = searcher.search(query,sort);
>>
>> And i had huge data about 13 millions of records
>> i am not sure y its giving outof memory exception and
>> no exception when no sorting is done
>> plz some one help me yar
>>
>> and also if to increase heap space how to increase it programatically i
>> had
>> command prompt
>> java -Xms<initial heap size> -Xmx<maximum heap size>
>> please.....
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16138043.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by sandyg <ga...@gmail.com>.
Thanks for the reply.
Actually am sorting on a specific field that is on keyword feild which is
unique
and i had 1 gb ram
markrmiller wrote:
>
> To sort on 13mil docs will take like at least 400 mb for the field
> cache. Thats if you only sort on one field...it can grow fast if you
> allow multi field sorting.
>
> How much RAM are you giving your app?
>
> sandyg wrote:
>> this is my search content
>>
>> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
>> Query query = parser.parse("1");
>>
>> Sort sort = new Sort(new SortField(sortField));
>> Hits hits = searcher.search(query,sort);
>>
>> And i had huge data about 13 millions of records
>> i am not sure y its giving outof memory exception and
>> no exception when no sorting is done
>> plz some one help me yar
>>
>> and also if to increase heap space how to increase it programatically i
>> had
>> command prompt
>> java -Xms<initial heap size> -Xmx<maximum heap size>
>> please.....
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16137836.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by Mark Miller <ma...@gmail.com>.
To sort on 13mil docs will take like at least 400 mb for the field
cache. Thats if you only sort on one field...it can grow fast if you
allow multi field sorting.
How much RAM are you giving your app?
sandyg wrote:
> this is my search content
>
> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
> Query query = parser.parse("1");
>
> Sort sort = new Sort(new SortField(sortField));
> Hits hits = searcher.search(query,sort);
>
> And i had huge data about 13 millions of records
> i am not sure y its giving outof memory exception and
> no exception when no sorting is done
> plz some one help me yar
>
> and also if to increase heap space how to increase it programatically i had
> command prompt
> java -Xms<initial heap size> -Xmx<maximum heap size>
> please.....
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by sandyg <ga...@gmail.com>.
How can i do sorting on the results i get (if already hits are there then
how to sort on the hits),instead of sorting on all the values b4 getting
results
Chris Lu wrote:
>
> This is because sorting will load all values in that sortFirled into
> memory.
>
> If it's an integer, you will need 4*N bytes, which is additional 52M for
> you.
>
> There is no programatical way to increase memory size.
>
> --
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes:
> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per request)
> got 2.6 Million Euro funding!
>
>
> On Tue, Mar 18, 2008 at 6:23 AM, sandyg <ga...@gmail.com> wrote:
>>
>> this is my search content
>>
>> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
>> Query query = parser.parse("1");
>>
>> Sort sort = new Sort(new SortField(sortField));
>> Hits hits = searcher.search(query,sort);
>>
>> And i had huge data about 13 millions of records
>> i am not sure y its giving outof memory exception and
>> no exception when no sorting is done
>> plz some one help me yar
>>
>> and also if to increase heap space how to increase it programatically i
>> had
>> command prompt
>> java -Xms<initial heap size> -Xmx<maximum heap size>
>> please.....
>> --
>> View this message in context:
>> http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16121128.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16137871.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the fields
Posted by Chris Lu <ch...@gmail.com>.
This is because sorting will load all values in that sortFirled into memory.
If it's an integer, you will need 4*N bytes, which is additional 52M for you.
There is no programatical way to increase memory size.
--
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer, a shopping comparison site, (anonymous per request)
got 2.6 Million Euro funding!
On Tue, Mar 18, 2008 at 6:23 AM, sandyg <ga...@gmail.com> wrote:
>
> this is my search content
>
> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
> Query query = parser.parse("1");
>
> Sort sort = new Sort(new SortField(sortField));
> Hits hits = searcher.search(query,sort);
>
> And i had huge data about 13 millions of records
> i am not sure y its giving outof memory exception and
> no exception when no sorting is done
> plz some one help me yar
>
> and also if to increase heap space how to increase it programatically i had
> command prompt
> java -Xms<initial heap size> -Xmx<maximum heap size>
> please.....
> --
> View this message in context: http://www.nabble.com/java.lang.OutOfMemoryError%3A-Java-heap-space-when-sorting-the-fields-tp16121128p16121128.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.OutOfMemoryError: Java heap space when sorting the
fields
Posted by Mark Miller <ma...@gmail.com>.
> Whoops...10 times to much there. more like 40 meg I think. A string
> sort could be a bit higher though, you also need to store all of terms
> to index into.
sandyg wrote:
> this is my search content
>
> QueryParser parser = new QueryParser("keyword",new StandardAnalyzer());
> Query query = parser.parse("1");
>
> Sort sort = new Sort(new SortField(sortField));
> Hits hits = searcher.search(query,sort);
>
> And i had huge data about 13 millions of records
> i am not sure y its giving outof memory exception and
> no exception when no sorting is done
> plz some one help me yar
>
> and also if to increase heap space how to increase it programatically i had
> command prompt
> java -Xms<initial heap size> -Xmx<maximum heap size>
> please.....
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org