You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by tgospodinov <sp...@gospodinov.net> on 2007/12/27 15:01:08 UTC
Improving jvm heap consumption when indexing
I am indexing a collection of 100,000+ sentences in memory, as part of a
client app. I tested the jvm heap consumption and it increases by about 40
megs. I tried indexing on disk (which I can't do in a production
environment) just to test the heap usage, and I came up with about 20-25
megs. I have another search alternative, which is much simpler, but doesn't
come close to Lucene, and it only consumes about 6-8 megs while indexing the
same number of entries. Is there a way to improve heap usage while indexing
in-memory?
I appreciate your help.
--
View this message in context: http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513435.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Improving jvm heap consumption when indexing
Posted by tgospodinov <sp...@gospodinov.net>.
I am using Lucene 2.2.0 on Java SE 1.6.0_03-b05. I will try reusing documents
and see how it'll affect heap. In the meantime, please let me know if
there's anything else I can do.
Thanks for the quick reply.
Grant Ingersoll-6 wrote:
>
> This question is better for java-user, but some questions for you:
>
> What version of Lucene are you using? The trunk now has the ability
> to reuse Tokens/Documents, etc. to save on allocations that may help
> reduce the amount of heap needed. The other thing to do is try
> profiling to see where the allocations are coming from and then report
> back to us. Also, info about your JVM, etc. would be helpful.
>
> -Grant
>
>
> On Dec 27, 2007, at 9:01 AM, tgospodinov wrote:
>
>>
>> I am indexing a collection of 100,000+ sentences in memory, as part
>> of a
>> client app. I tested the jvm heap consumption and it increases by
>> about 40
>> megs. I tried indexing on disk (which I can't do in a production
>> environment) just to test the heap usage, and I came up with about
>> 20-25
>> megs. I have another search alternative, which is much simpler, but
>> doesn't
>> come close to Lucene, and it only consumes about 6-8 megs while
>> indexing the
>> same number of entries. Is there a way to improve heap usage while
>> indexing
>> in-memory?
>>
>> I appreciate your help.
>> --
>> View this message in context:
>> http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513435.html
>> Sent from the Lucene - Java Developer mailing list archive at
>> Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513674.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Improving jvm heap consumption when indexing
Posted by Grant Ingersoll <gs...@apache.org>.
You will also want to reuse the Token using Lucene 2.3-dev (i.e the
latest from Subversion.)
-Grant
On Dec 27, 2007, at 10:26 AM, tgospodinov wrote:
>
> Reusing documents and invoking garbage collection slowed
> performance, but
> improved heap usage big time (From an average of 20 megs+ to about 12
> megs+). I am wondering if there's anything else that can be done?
>
>
> Grant Ingersoll-6 wrote:
>>
>> This question is better for java-user, but some questions for you:
>>
>> What version of Lucene are you using? The trunk now has the ability
>> to reuse Tokens/Documents, etc. to save on allocations that may help
>> reduce the amount of heap needed. The other thing to do is try
>> profiling to see where the allocations are coming from and then
>> report
>> back to us. Also, info about your JVM, etc. would be helpful.
>>
>> -Grant
>>
>>
>> On Dec 27, 2007, at 9:01 AM, tgospodinov wrote:
>>
>>>
>>> I am indexing a collection of 100,000+ sentences in memory, as part
>>> of a
>>> client app. I tested the jvm heap consumption and it increases by
>>> about 40
>>> megs. I tried indexing on disk (which I can't do in a production
>>> environment) just to test the heap usage, and I came up with about
>>> 20-25
>>> megs. I have another search alternative, which is much simpler, but
>>> doesn't
>>> come close to Lucene, and it only consumes about 6-8 megs while
>>> indexing the
>>> same number of entries. Is there a way to improve heap usage while
>>> indexing
>>> in-memory?
>>>
>>> I appreciate your help.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513435.html
>>> Sent from the Lucene - Java Developer mailing list archive at
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>> --------------------------
>> Grant Ingersoll
>> http://lucene.grantingersoll.com
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14514440.html
> Sent from the Lucene - Java Developer mailing list archive at
> Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Improving jvm heap consumption when indexing
Posted by tgospodinov <sp...@gospodinov.net>.
Reusing documents and invoking garbage collection slowed performance, but
improved heap usage big time (From an average of 20 megs+ to about 12
megs+). I am wondering if there's anything else that can be done?
Grant Ingersoll-6 wrote:
>
> This question is better for java-user, but some questions for you:
>
> What version of Lucene are you using? The trunk now has the ability
> to reuse Tokens/Documents, etc. to save on allocations that may help
> reduce the amount of heap needed. The other thing to do is try
> profiling to see where the allocations are coming from and then report
> back to us. Also, info about your JVM, etc. would be helpful.
>
> -Grant
>
>
> On Dec 27, 2007, at 9:01 AM, tgospodinov wrote:
>
>>
>> I am indexing a collection of 100,000+ sentences in memory, as part
>> of a
>> client app. I tested the jvm heap consumption and it increases by
>> about 40
>> megs. I tried indexing on disk (which I can't do in a production
>> environment) just to test the heap usage, and I came up with about
>> 20-25
>> megs. I have another search alternative, which is much simpler, but
>> doesn't
>> come close to Lucene, and it only consumes about 6-8 megs while
>> indexing the
>> same number of entries. Is there a way to improve heap usage while
>> indexing
>> in-memory?
>>
>> I appreciate your help.
>> --
>> View this message in context:
>> http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513435.html
>> Sent from the Lucene - Java Developer mailing list archive at
>> Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14514440.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Improving jvm heap consumption when indexing
Posted by Grant Ingersoll <gs...@apache.org>.
This question is better for java-user, but some questions for you:
What version of Lucene are you using? The trunk now has the ability
to reuse Tokens/Documents, etc. to save on allocations that may help
reduce the amount of heap needed. The other thing to do is try
profiling to see where the allocations are coming from and then report
back to us. Also, info about your JVM, etc. would be helpful.
-Grant
On Dec 27, 2007, at 9:01 AM, tgospodinov wrote:
>
> I am indexing a collection of 100,000+ sentences in memory, as part
> of a
> client app. I tested the jvm heap consumption and it increases by
> about 40
> megs. I tried indexing on disk (which I can't do in a production
> environment) just to test the heap usage, and I came up with about
> 20-25
> megs. I have another search alternative, which is much simpler, but
> doesn't
> come close to Lucene, and it only consumes about 6-8 megs while
> indexing the
> same number of entries. Is there a way to improve heap usage while
> indexing
> in-memory?
>
> I appreciate your help.
> --
> View this message in context: http://www.nabble.com/Improving-jvm-heap-consumption-when-indexing-tp14513435p14513435.html
> Sent from the Lucene - Java Developer mailing list archive at
> Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Improving jvm heap consumption when indexing
Posted by robert engels <re...@ix.netcom.com>.
Did you try running with -Xmx8m ? If you don't limit the heap, the
JVM will use as much as you allow it...
On Dec 27, 2007, at 8:01 AM, tgospodinov wrote:
>
> I am indexing a collection of 100,000+ sentences in memory, as part
> of a
> client app. I tested the jvm heap consumption and it increases by
> about 40
> megs. I tried indexing on disk (which I can't do in a production
> environment) just to test the heap usage, and I came up with about
> 20-25
> megs. I have another search alternative, which is much simpler, but
> doesn't
> come close to Lucene, and it only consumes about 6-8 megs while
> indexing the
> same number of entries. Is there a way to improve heap usage while
> indexing
> in-memory?
>
> I appreciate your help.
> --
> View this message in context: http://www.nabble.com/Improving-jvm-
> heap-consumption-when-indexing-tp14513435p14513435.html
> Sent from the Lucene - Java Developer mailing list archive at
> Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org