You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by stockii <st...@googlemail.com> on 2011/03/29 14:22:24 UTC
how to start GarbageCollector
Hello,
my problem is, that after a full-import solr reserved all of my RAM and my
delta-imports need about 1 hour for less than 5000 small documents.
How can i start GarbageCollector to get the RAM back ?
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2748080.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
why is solr copy my complete index to somewhere when i start an delta-import?
i copy one core, start an full-import from 35Million docs and then start an
delta-import from the last hour (~2000Docs).
dih/solr need start to copy the hole index... why ? i think he is copy the
index, because my hdd-space starts to increase imediatly ...
my live core ended a delta in 5-10 seconds !?!?!?!?!?!?
i run jconsole during this time, what say it to me ?
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2783923.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
okay, i thougt that heap isnt the problem ...
but, what should i look for in jconsole ? what say it to me ?
i understand the monitoring sense not.
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2752807.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by Markus Jelsma <ma...@openindex.io>.
Stockii, don't be sad but read about JVM memory usage and collection.
Increasing and decreasing memory consumption is normal, you would only worry
if the left bar reaches 100% and stays at 100%.
Start overhere:
http://download.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html
But again, memory is most likely _not_ your problem here!
Cheers
> jconsole: heap: 100 % =(((
> what can i do ?
> http://lucene.472066.n3.nabble.com/file/n2752697/heap.png
>
>
> -----
> ------------------------------- System
> ----------------------------------------
>
> One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
> 1 Core with 31 Million Documents other Cores < 100.000
>
> - Solr1 for Search-Requests - commit every Minute - 5GB Xmx
> - Solr2 for Update-Request - delta every Minute - 4GB Xmx
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080
> p2752697.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
when a delta-import is startet my heap jumped to 100% ... every minute ...
when GC is running heap is near zero.
how can i optimize this ?
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2752707.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
jconsole: heap: 100 % =(((
what can i do ?
http://lucene.472066.n3.nabble.com/file/n2752697/heap.png
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2752697.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
okay, i installed an monitor, jconsole and jvisualvm. how can i see with
this, where my probem is ?
what data are needed ? :/
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2748421.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by Markus Jelsma <ma...@openindex.io>.
Unix tools won't show heap usage statistics. Please use tools that come with
your JVM such as jps, jtop, jstat or setup monitoring over JMX to get a good
picture.
All aside, RAM is most likely not your problem.
> i run an full-import via DIH, 35 Million Documents, i dont restart solr. my
> cronjob start automaticly an delta. if i restart solr, delta obtain in ~10
> seconds ...
>
>
> "free -m" show me how many RAM is beeing used and with "top". the server is
> only for solr, so no other processes are using my RAM.
>
> -----
> ------------------------------- System
> ----------------------------------------
>
> One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
> 1 Core with 31 Million Documents other Cores < 100.000
>
> - Solr1 for Search-Requests - commit every Minute - 5GB Xmx
> - Solr2 for Update-Request - delta every Minute - 4GB Xmx
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080
> p2748134.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by stockii <st...@googlemail.com>.
i run an full-import via DIH, 35 Million Documents, i dont restart solr. my
cronjob start automaticly an delta. if i restart solr, delta obtain in ~10
seconds ...
"free -m" show me how many RAM is beeing used and with "top". the server is
only for solr, so no other processes are using my RAM.
-----
------------------------------- System ----------------------------------------
One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
1 Core with 31 Million Documents other Cores < 100.000
- Solr1 for Search-Requests - commit every Minute - 5GB Xmx
- Solr2 for Update-Request - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2748134.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to start GarbageCollector
Posted by Erick Erickson <er...@gmail.com>.
I doubt this is your issue, the garbage collector will run automatically
at need.
What happens if you do a full import, stop and restart your Solr server
and then try the delta? If the delta takes an hour then it has nothing to
do with garbage collection.
What are you importing from? I'd suspect that your problem is more
along the lines that your extraction process somehow takes a long
time to just extract the data and Solr is a red herring...
Best
Erick
On Tue, Mar 29, 2011 at 8:22 AM, stockii <st...@googlemail.com> wrote:
> Hello,
>
> my problem is, that after a full-import solr reserved all of my RAM and my
> delta-imports need about 1 hour for less than 5000 small documents.
>
> How can i start GarbageCollector to get the RAM back ?
>
> -----
> ------------------------------- System ----------------------------------------
>
> One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
> 1 Core with 31 Million Documents other Cores < 100.000
>
> - Solr1 for Search-Requests - commit every Minute - 5GB Xmx
> - Solr2 for Update-Request - delta every Minute - 4GB Xmx
> --
> View this message in context: http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080p2748080.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: how to start GarbageCollector
Posted by Markus Jelsma <ma...@openindex.io>.
I seriously doubt heap usage is actually your problem. Usually a garbage
collector is running, if it (somehow) doesn't you will definately run out of
memory some time. Where did you check memory usage?
> Hello,
>
> my problem is, that after a full-import solr reserved all of my RAM and my
> delta-imports need about 1 hour for less than 5000 small documents.
>
> How can i start GarbageCollector to get the RAM back ?
>
> -----
> ------------------------------- System
> ----------------------------------------
>
> One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
> 1 Core with 31 Million Documents other Cores < 100.000
>
> - Solr1 for Search-Requests - commit every Minute - 5GB Xmx
> - Solr2 for Update-Request - delta every Minute - 4GB Xmx
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/how-to-start-GarbageCollector-tp2748080
> p2748080.html Sent from the Solr - User mailing list archive at Nabble.com.