You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Schoy <he...@gmx.de> on 2011/06/17 14:10:18 UTC
Indexing-speed issues (chart included)
Hi,
If I start indexing documents it getting slower the more documents were
added without commiting and optimizing:
http://imageshack.us/photo/my-images/695/solrchart.png/
I've changed the mergeFactor from 10 to 30, changed maxDocs (100,1000,10000)
but it always getting slower the more documents were added.
If I'm using elasticsearch which is also based on lucene I'm getting
constant indexing rates (without commiting and optimizing too)
Does anybody know whats wrong?
Re: Indexing-speed issues (chart included)
Posted by Erick Erickson <er...@gmail.com>.
No, generally this isn't what I'd expect. There will be periodic
slowdowns when segments are flushed (I'm assuming
you're not using trunk, there have been speedups here, see:
http://blog.jteam.nl/2011/04/01/gimme-all-resources-you-have-i-can-use-them/)
Does your config have any <autocommit> parameters set? You
might be committing without knowing you are.....
Best
Erick
On Fri, Jun 17, 2011 at 8:34 AM, Mark Schoy <he...@gmx.de> wrote:
> Sorry, here are some details:
>
> requestHandler: XmlUpdateRequesetHandler
> protocol: http (10 concurrend threads)
> document: 1kb size, 15 fields
>
> cpu load: 20%
> memory usage: 50%
>
> But generally speaking, is that normal or must be something wrong with my
> configuration, ...
>
> 2011/6/17 Erick Erickson <er...@gmail.com>
>
>> Well, it's kinda hard to say anything pertinent with so little
>> information. How are you indexing things? What kind of documents?
>> How are you feeding docs to Solr?
>>
>> You might review:
>> http://wiki.apache.org/solr/UsingMailingLists
>>
>> Best
>> Erick
>>
>> On Fri, Jun 17, 2011 at 8:10 AM, Mark Schoy <he...@gmx.de> wrote:
>> > Hi,
>> >
>> > If I start indexing documents it getting slower the more documents were
>> > added without commiting and optimizing:
>> >
>> > http://imageshack.us/photo/my-images/695/solrchart.png/
>> >
>> > I've changed the mergeFactor from 10 to 30, changed maxDocs
>> (100,1000,10000)
>> > but it always getting slower the more documents were added.
>> > If I'm using elasticsearch which is also based on lucene I'm getting
>> > constant indexing rates (without commiting and optimizing too)
>> >
>> > Does anybody know whats wrong?
>> >
>>
>
Re: Indexing-speed issues (chart included)
Posted by Mark Schoy <he...@gmx.de>.
Sorry, here are some details:
requestHandler: XmlUpdateRequesetHandler
protocol: http (10 concurrend threads)
document: 1kb size, 15 fields
cpu load: 20%
memory usage: 50%
But generally speaking, is that normal or must be something wrong with my
configuration, ...
2011/6/17 Erick Erickson <er...@gmail.com>
> Well, it's kinda hard to say anything pertinent with so little
> information. How are you indexing things? What kind of documents?
> How are you feeding docs to Solr?
>
> You might review:
> http://wiki.apache.org/solr/UsingMailingLists
>
> Best
> Erick
>
> On Fri, Jun 17, 2011 at 8:10 AM, Mark Schoy <he...@gmx.de> wrote:
> > Hi,
> >
> > If I start indexing documents it getting slower the more documents were
> > added without commiting and optimizing:
> >
> > http://imageshack.us/photo/my-images/695/solrchart.png/
> >
> > I've changed the mergeFactor from 10 to 30, changed maxDocs
> (100,1000,10000)
> > but it always getting slower the more documents were added.
> > If I'm using elasticsearch which is also based on lucene I'm getting
> > constant indexing rates (without commiting and optimizing too)
> >
> > Does anybody know whats wrong?
> >
>
Re: Indexing-speed issues (chart included)
Posted by Mathias Hodler <ma...@gmail.com>.
Sorry, here are some details:
requestHandler: XmlUpdateRequesetHandler
protocol: http (10 concurrend threads)
document: 1kb size, 15 fields
cpu load: 20%
memory usage: 50%
But generally speaking, is that normal or must be something wrong with my
configuration, ...
2011/6/17 Erick Erickson <er...@gmail.com>
> Well, it's kinda hard to say anything pertinent with so little
> information. How are you indexing things? What kind of documents?
> How are you feeding docs to Solr?
>
> You might review:
> http://wiki.apache.org/solr/UsingMailingLists
>
> Best
> Erick
>
> On Fri, Jun 17, 2011 at 8:10 AM, Mark Schoy <he...@gmx.de> wrote:
> > Hi,
> >
> > If I start indexing documents it getting slower the more documents were
> > added without commiting and optimizing:
> >
> > http://imageshack.us/photo/my-images/695/solrchart.png/
> >
> > I've changed the mergeFactor from 10 to 30, changed maxDocs
> (100,1000,10000)
> > but it always getting slower the more documents were added.
> > If I'm using elasticsearch which is also based on lucene I'm getting
> > constant indexing rates (without commiting and optimizing too)
> >
> > Does anybody know whats wrong?
> >
>
Re: Indexing-speed issues (chart included)
Posted by Erick Erickson <er...@gmail.com>.
Well, it's kinda hard to say anything pertinent with so little
information. How are you indexing things? What kind of documents?
How are you feeding docs to Solr?
You might review:
http://wiki.apache.org/solr/UsingMailingLists
Best
Erick
On Fri, Jun 17, 2011 at 8:10 AM, Mark Schoy <he...@gmx.de> wrote:
> Hi,
>
> If I start indexing documents it getting slower the more documents were
> added without commiting and optimizing:
>
> http://imageshack.us/photo/my-images/695/solrchart.png/
>
> I've changed the mergeFactor from 10 to 30, changed maxDocs (100,1000,10000)
> but it always getting slower the more documents were added.
> If I'm using elasticsearch which is also based on lucene I'm getting
> constant indexing rates (without commiting and optimizing too)
>
> Does anybody know whats wrong?
>