Solr and Lucene Indexing Performance

[Jae Joo <ja...@gmail.com>]

Hi,

I have 6 millions article to be indexed by Solr and do need your
recommendation.

I do need to parse and generate the Solr based xml file to post it. How
about to use Lucene directly?
I have short testing, it looks like Sola based indexing is faster than
direct indexing through Lucene.

Am I did something wrong and/or does Solr use multiple threading or
something else to get the good indexing performance?

Thanks

Jae Joo

Re: Solr and Lucene Indexing Performance

[Mike Klaas <mi...@gmail.com>]

On 2-Nov-07, at 11:41 AM, Jae Joo wrote:

> Hi,
>
> I have 6 millions article to be indexed by Solr and do need your
> recommendation.
>
> I do need to parse and generate the Solr based xml file to post it.  
> How
> about to use Lucene directly?
> I have short testing, it looks like Sola based indexing is faster than
> direct indexing through Lucene.

I wouldn't recommend that.  If you use persistent connections,  
multiple threads and >1 docs/update you should achieve comparable  
performance (about 10docs/request is about the right balance for web- 
sized docs).

If you want index directly, use embedded Solr, not Lucene directly  
(see the wiki).

> Am I did something wrong and/or does Solr use multiple threading or
> something else to get the good indexing performance?

It does use multiple threads if you connect to Solr using multiple  
threads.  But it doesn't do it behind the scenes if you aren't using  
multiple threads.

Some possible differences:

1. Solr has more aggressive default buffering settings  
(maxBufferedDocs, mergeFactor)
2. solr trunk (if that is what you are using) is using a more recent  
version of Lucene than the released 2.2

-Mike