IndexWrite in Lucene/Solr 3.5 is slower?

[Ramprakash Ramamoorthy <yo...@gmail.com>]

We are upgrading our search infrastructure from Lucene 2.3.1 to Lucene 3.5.
I am in the process of load testing and I could find that Lucene 2.3.1
could index 32,000 docs per second, whereas Lucene 3.5 could index only
around 17,000 docs per second.

Indeed, both of them use the standard analyzer and the default settings. Is
3.5 slower because it indexes more details and thereby resulting in a
faster search? Ours is a log management product and the speed of indexing
is highly important.

Ok, cutting the long story short, will the slower indexing of 3.5 result in
a higher search speed?, if not, what else should I fine tune to improve the
indexing speed?

-- 
With Thanks and Regards,
Ramprakash Ramamoorthy,
Engineer Trainee,
Zoho Corporation.
+91 9626975420

Re: IndexWrite in Lucene/Solr 3.5 is slower?

[Torsten Krah <tk...@fachschaft.imn.htwk-leipzig.de>]

May be related to https://issues.apache.org/jira/browse/LUCENE-3418
which does ensure things are really written; if you do commit very
often, you may see this sort of performance loss (at least me did in my
junit test where i do commit very often and 3.3 switch to 3.4 really
hurts here at test time - but its ok for tests to take longer because
the real app does use batch commits).

You can try to use solr 3.3.x and see how it works against 3.4.0 (which
does include the fix for 3418) if you want to find out if this is
related to #3418.

HTH

Torsten

Re: IndexWrite in Lucene/Solr 3.5 is slower?

[Ramprakash Ramamoorthy <yo...@gmail.com>]

On Fri, Jun 15, 2012 at 12:50 PM, Ramprakash Ramamoorthy <
youngestachiever@gmail.com> wrote:

>
>
> On Fri, Jun 15, 2012 at 12:20 PM, pravesh <su...@yahoo.com> wrote:
>
>> BTW, Have you changed the MergePolicy & MergeScheduler settings also?
>> Since
>> Lucene 3.x/3.5 onwards,
>> there have been new MergePolicy & MergeScheduler implementations
>> available,
>> like TieredMergePolicy & ConcurrentMergeScheduler.
>>
>> Regards
>> Pravesh
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/IndexWrite-in-Lucene-Solr-3-5-is-slower-tp3989764p3989768.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
> Thanks for the reply Pravesh. Yes I initially used the default
>  TieredMergePolicy and later set the merge policy in both the versions to
> LogByteSizeMergePolicy, in order to maintain congruence. But still Lucene
> 3.5 lagged behind by 2X approx.
>
>
> --
> With Thanks and Regards,
> Ramprakash Ramamoorthy,
> Engineer Trainee,
> Zoho Corporation.
> +91 9626975420
>
>
Can someone help me with this please?

-- 
With Thanks and Regards,
Ramprakash Ramamoorthy,
Engineer Trainee,
Zoho Corporation.
+91 9626975420

Re: IndexWrite in Lucene/Solr 3.5 is slower?

[Ramprakash Ramamoorthy <yo...@gmail.com>]

On Fri, Jun 15, 2012 at 12:20 PM, pravesh <su...@yahoo.com> wrote:

> BTW, Have you changed the MergePolicy & MergeScheduler settings also? Since
> Lucene 3.x/3.5 onwards,
> there have been new MergePolicy & MergeScheduler implementations available,
> like TieredMergePolicy & ConcurrentMergeScheduler.
>
> Regards
> Pravesh
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/IndexWrite-in-Lucene-Solr-3-5-is-slower-tp3989764p3989768.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Thanks for the reply Pravesh. Yes I initially used the default
 TieredMergePolicy and later set the merge policy in both the versions to
LogByteSizeMergePolicy, in order to maintain congruence. But still Lucene
3.5 lagged behind by 2X approx.

-- 
With Thanks and Regards,
Ramprakash Ramamoorthy,
Engineer Trainee,
Zoho Corporation.
+91 9626975420

Re: IndexWrite in Lucene/Solr 3.5 is slower?

[pravesh <su...@yahoo.com>]

BTW, Have you changed the MergePolicy & MergeScheduler settings also? Since
Lucene 3.x/3.5 onwards,
there have been new MergePolicy & MergeScheduler implementations available,
like TieredMergePolicy & ConcurrentMergeScheduler.

Regards
Pravesh

--
View this message in context: http://lucene.472066.n3.nabble.com/IndexWrite-in-Lucene-Solr-3-5-is-slower-tp3989764p3989768.html
Sent from the Solr - User mailing list archive at Nabble.com.