You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Willem Basson <wi...@gmail.com> on 2011/10/07 10:04:25 UTC
Slowdown on adds with Solr 3.4 compared to 1.4
Hi there
We are currently moving from Solr 1.4 to 3.4 and we are seeing a few issues
with adding documents.
We do a delete by query and then do a lot of adds, about 100k before we do a
commit and optimise.
With 1.4 this was all fine, not super quick but didn't see any problems.
With 3.4 the rate of adding documents seriously degrades. For our one index
at about 80% it severely slows down but struggles and completes.
For the other index which has quite a few more fields (up to 6000+ for some
documents) it slows down at about 20%.
If we do periodic commits then we don't see the slowdown, but that causes us
some other issues with replication etc. and while we can go down that route
if we really must we would like to know what has changed from 1.4 to 3.4 to
cause this behaviour. I have tried changing the LuceneMatchVersion to
LUCENE_34 and upped to memory from 2GB to 4GB on a machine with 8GB ram but
it really doesn't make any difference to the behaviour. Don't see any errors
in the log files.
Any ideas of what we could try to diagnose or fix the problem?
--
Willem Basson
Re: Slowdown on adds with Solr 3.4 compared to 1.4
Posted by Willem Basson <wi...@gmail.com>.
Hi there
Yes I have tried that, and it makes a small difference but still getting the
slowdown, just slightly later.
No hints from the logs about the slowdown, no errors or useful info in there
even if I set all the logging on.
Willem
On Fri, Oct 7, 2011 at 2:38 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Have you tried to do a commit after the deleteByQuery only?
> Also, what seems to cause the slowdown? Any hints from the logs?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 7. okt. 2011, at 10:04, Willem Basson wrote:
>
> > Hi there
> >
> > We are currently moving from Solr 1.4 to 3.4 and we are seeing a few
> issues
> > with adding documents.
> > We do a delete by query and then do a lot of adds, about 100k before we
> do a
> > commit and optimise.
> > With 1.4 this was all fine, not super quick but didn't see any problems.
> > With 3.4 the rate of adding documents seriously degrades. For our one
> index
> > at about 80% it severely slows down but struggles and completes.
> > For the other index which has quite a few more fields (up to 6000+ for
> some
> > documents) it slows down at about 20%.
> >
> > If we do periodic commits then we don't see the slowdown, but that causes
> us
> > some other issues with replication etc. and while we can go down that
> route
> > if we really must we would like to know what has changed from 1.4 to 3.4
> to
> > cause this behaviour. I have tried changing the LuceneMatchVersion to
> > LUCENE_34 and upped to memory from 2GB to 4GB on a machine with 8GB ram
> but
> > it really doesn't make any difference to the behaviour. Don't see any
> errors
> > in the log files.
> >
> > Any ideas of what we could try to diagnose or fix the problem?
> >
> > --
> > Willem Basson
>
>
--
Willem Basson
Re: Slowdown on adds with Solr 3.4 compared to 1.4
Posted by Willem Basson <wi...@gmail.com>.
Hi Jan
Thanks for getting back to me. Here's the details:
JVM: jdk1.6.0_27
App Server: Tomcat 7
OS: Centos x86_64 5.4
RAM: 8GB
JVM RAM: 4GB
JVM ARGS: -Xms4G -Xmx4G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:NewRatio=3 -XX:PermSize=128M -XX:MaxPermSize=256M
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Xloggc:/blah/blah....
I haven't made any changes to the Solr config files.
I have tried adding this:
<luceneMatchVersion>LUCENE_34</luceneMatchVersion>
but it makes no difference. Otherwise same as it always was.
We do a delete by query to delete all the documents and then post data as
xml in batches to solr before we do an optimize.
We have replication set to replicate on optimize.
As I said this was working fine with 1.4 but not with 3.4.
Regarding visualvm, I've been using that to look at the memory usage for a
while, but don't see anything weird. The JVM memory management looks ok, the
heap saw-tooths nicely and seems to gc OK. There's no correlation with the
garbage collection and the slowdown of adds.
Willem
On Fri, Oct 7, 2011 at 11:17 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Could you post more info about your environment?
>
> OS, JVM version, Total RAM and how much allocated to JVM? JVM options such
> as GC settings, other applications running on same box? document type and
> size, size of your "index" folder on disk, schema & solrconfig changes (have
> you migrated schema since 1.4?), what have you changed in solrconfig.xml?
> What method/API/UpdateHandler do you use to feed documents?
>
> You should also try to use Java VisualVM (
> http://visualvm.java.net/index.html) and connect to your running JVM to
> analyze heap size, GC activity and profile mem/cpu usage. Report back what
> you find.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 7. okt. 2011, at 15:38, Jan Høydahl wrote:
>
> > Hi,
> >
> > Have you tried to do a commit after the deleteByQuery only?
> > Also, what seems to cause the slowdown? Any hints from the logs?
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> > Solr Training - www.solrtraining.com
> >
> > On 7. okt. 2011, at 10:04, Willem Basson wrote:
> >
> >> Hi there
> >>
> >> We are currently moving from Solr 1.4 to 3.4 and we are seeing a few
> issues
> >> with adding documents.
> >> We do a delete by query and then do a lot of adds, about 100k before we
> do a
> >> commit and optimise.
> >> With 1.4 this was all fine, not super quick but didn't see any problems.
> >> With 3.4 the rate of adding documents seriously degrades. For our one
> index
> >> at about 80% it severely slows down but struggles and completes.
> >> For the other index which has quite a few more fields (up to 6000+ for
> some
> >> documents) it slows down at about 20%.
> >>
> >> If we do periodic commits then we don't see the slowdown, but that
> causes us
> >> some other issues with replication etc. and while we can go down that
> route
> >> if we really must we would like to know what has changed from 1.4 to 3.4
> to
> >> cause this behaviour. I have tried changing the LuceneMatchVersion to
> >> LUCENE_34 and upped to memory from 2GB to 4GB on a machine with 8GB ram
> but
> >> it really doesn't make any difference to the behaviour. Don't see any
> errors
> >> in the log files.
> >>
> >> Any ideas of what we could try to diagnose or fix the problem?
> >>
> >> --
> >> Willem Basson
> >
>
Re: Slowdown on adds with Solr 3.4 compared to 1.4
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Could you post more info about your environment?
OS, JVM version, Total RAM and how much allocated to JVM? JVM options such as GC settings, other applications running on same box? document type and size, size of your "index" folder on disk, schema & solrconfig changes (have you migrated schema since 1.4?), what have you changed in solrconfig.xml? What method/API/UpdateHandler do you use to feed documents?
You should also try to use Java VisualVM (http://visualvm.java.net/index.html) and connect to your running JVM to analyze heap size, GC activity and profile mem/cpu usage. Report back what you find.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 7. okt. 2011, at 15:38, Jan Høydahl wrote:
> Hi,
>
> Have you tried to do a commit after the deleteByQuery only?
> Also, what seems to cause the slowdown? Any hints from the logs?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 7. okt. 2011, at 10:04, Willem Basson wrote:
>
>> Hi there
>>
>> We are currently moving from Solr 1.4 to 3.4 and we are seeing a few issues
>> with adding documents.
>> We do a delete by query and then do a lot of adds, about 100k before we do a
>> commit and optimise.
>> With 1.4 this was all fine, not super quick but didn't see any problems.
>> With 3.4 the rate of adding documents seriously degrades. For our one index
>> at about 80% it severely slows down but struggles and completes.
>> For the other index which has quite a few more fields (up to 6000+ for some
>> documents) it slows down at about 20%.
>>
>> If we do periodic commits then we don't see the slowdown, but that causes us
>> some other issues with replication etc. and while we can go down that route
>> if we really must we would like to know what has changed from 1.4 to 3.4 to
>> cause this behaviour. I have tried changing the LuceneMatchVersion to
>> LUCENE_34 and upped to memory from 2GB to 4GB on a machine with 8GB ram but
>> it really doesn't make any difference to the behaviour. Don't see any errors
>> in the log files.
>>
>> Any ideas of what we could try to diagnose or fix the problem?
>>
>> --
>> Willem Basson
>
Re: Slowdown on adds with Solr 3.4 compared to 1.4
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Have you tried to do a commit after the deleteByQuery only?
Also, what seems to cause the slowdown? Any hints from the logs?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 7. okt. 2011, at 10:04, Willem Basson wrote:
> Hi there
>
> We are currently moving from Solr 1.4 to 3.4 and we are seeing a few issues
> with adding documents.
> We do a delete by query and then do a lot of adds, about 100k before we do a
> commit and optimise.
> With 1.4 this was all fine, not super quick but didn't see any problems.
> With 3.4 the rate of adding documents seriously degrades. For our one index
> at about 80% it severely slows down but struggles and completes.
> For the other index which has quite a few more fields (up to 6000+ for some
> documents) it slows down at about 20%.
>
> If we do periodic commits then we don't see the slowdown, but that causes us
> some other issues with replication etc. and while we can go down that route
> if we really must we would like to know what has changed from 1.4 to 3.4 to
> cause this behaviour. I have tried changing the LuceneMatchVersion to
> LUCENE_34 and upped to memory from 2GB to 4GB on a machine with 8GB ram but
> it really doesn't make any difference to the behaviour. Don't see any errors
> in the log files.
>
> Any ideas of what we could try to diagnose or fix the problem?
>
> --
> Willem Basson