Indexing a lot of documents?

[Jordan Hayes <jm...@speakeasy.net>]

New user here, and I ran into a problem trying to load a lot of 
documents (~900k).  I tried to load them all at once, which seemed to 
run for a long time and then finally crap out with "Too many open files" 
... so I read in an FAQ that "about 100" might be a good number.  I 
split my documents up and added <commit/> to the end of each batch, and 
got about 10k into it before getting that error again.

Am I just doing something wrong?  And: is there a way to just hand the 
XML file to Solr without having to POST it?

Thanks,

/jordan 

Re: Indexing a lot of documents?

[Mike Klaas <mi...@gmail.com>]

On 1-Jun-07, at 11:11 PM, Jordan Hayes wrote:

>
>> Lucene sometimes just requires many file descriptors (this will   
>> be somewhat alleviated with Solr 1.2).
>
> Is there a way to find out how many is "many" ...?

for each segment, 7 + num indexed fields per segment.  There should  
be log_{base mergefactor}(numDocs) * mergeFactor segments,  
approximately.

-Mike

Re: Indexing a lot of documents?

[Jordan Hayes <jm...@speakeasy.net>]

> I'm not clear exactly what you mean by batches.

What I'm doing is:

  <add>
    <doc>  <-- #1
    [...]
    <doc>  <-- #100
  </add><commit/>

So that's 100, in 1 HTTP POST.

> Lucene sometimes just requires many file descriptors (this will  
> be somewhat alleviated with Solr 1.2).

Is there a way to find out how many is "many" ...?

/jordan

Re: Indexing a lot of documents?

[Yonik Seeley <yo...@apache.org>]

On 6/1/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> On Jun 1, 2007, at 10:47 PM, Mike Klaas wrote:
> >> Am I just doing something wrong?
> >
> > No.  Lucene sometimes just requires many file descriptors (this
> > will be somewhat alleviated with Solr 1.2).  I suggest upping the
> > open file limit (I upped mine from 1024 to 45000 to handle huge
> > indices).  You can alleviate this by reducing the mergeFactor, but
> > this can impact indexing performance.
>
> Another thing to do which will definitely keep file handles down is
> to set to the compound index format.  That setting is in solrconfig.xml

That should be less necessary when Solr 1.2 comes out (next week, I promise ;-)
There are now 8 files per segment instead of 7 + num_indexed_fields

-Yonik

Re: Indexing a lot of documents?

[Erik Hatcher <er...@ehatchersolutions.com>]

On Jun 1, 2007, at 10:47 PM, Mike Klaas wrote:
>> Am I just doing something wrong?
>
> No.  Lucene sometimes just requires many file descriptors (this  
> will be somewhat alleviated with Solr 1.2).  I suggest upping the  
> open file limit (I upped mine from 1024 to 45000 to handle huge  
> indices).  You can alleviate this by reducing the mergeFactor, but  
> this can impact indexing performance.

Another thing to do which will definitely keep file handles down is  
to set to the compound index format.  That setting is in solrconfig.xml

	Erik

Re: Indexing a lot of documents?

[Mike Klaas <mi...@gmail.com>]

On 1-Jun-07, at 6:35 PM, Jordan Hayes wrote:

> New user here, and I ran into a problem trying to load a lot of  
> documents (~900k).  I tried to load them all at once, which seemed  
> to run for a long time and then finally crap out with "Too many  
> open files" ... so I read in an FAQ that "about 100" might be a  
> good number.  I split my documents up and added <commit/> to the  
> end of each batch, and got about 10k into it before getting that  
> error again.

I'm not clear exactly what you mean by batches.  There are two types:

1. batches of documents sent in a single <add> command to Solr.  Good  
values are between 10 and 100, depending on document size.  It is  
mostly about reducing http overhead (which is small regardless), so  
it is very quickly pointless to increase this number.  Try persistent  
HTTP connections
2. batches of docs sent between <commit/>.   In theory unlimited, but  
I once ran into a problem that I could not reliably reproduce when  
<commit/>ing 4m docs.  It occurred under tightish memory conditions  
(for 4m docs) and I've since made a change to Solr's deleted docs  
algo which should optimize the io in such cases.  In any case,  
<commit/>ing every 300-400k docs would not hurt.  I would not ever  
commit as frequently as 100 docs, unless there was query timeliness  
requirements.

> Am I just doing something wrong?

No.  Lucene sometimes just requires many file descriptors (this will  
be somewhat alleviated with Solr 1.2).  I suggest upping the open  
file limit (I upped mine from 1024 to 45000 to handle huge indices).   
You can alleviate this by reducing the mergeFactor, but this can  
impact indexing performance.

> And: is there a way to just hand the XML file to Solr without  
> having to POST it?

No, but POST'ing shouldn't be a bottleneck.

-Mike