Re: Solrindex adding documents in small chunks

[Bai Shen <ba...@gmail.com>]

Yep, I was aware of the solr.commit.size setting.  However, I'm not even
getting commits that big.  As my previous example shows, half of my commits
are single digit commits.  90% of them never break 100.

I'm trying to figure out why they're so small.


On Tue, Apr 30, 2013 at 10:45 AM, Canan GİRGİN <ca...@gmail.com>wrote:

> >Secondly, it's adding my documents in small chunks.  I was fetching in 100
> >document cycles and when I run solrindex I get messages such as the
> >following.
>
> In nutch-default.xml file, there is a parameter about solr commit size.
> If you want to add big chunks to index, you should increase this paremeter:
>
>
> <property>
>   <name>solr.commit.size</name>
>   <value>250</value>
>   <description>
>   Defines the number of documents to send to Solr in a single update batch.
>   Decrease when handling very large documents to prevent Nutch from running
>   out of memory. NOTE: It does not explicitly trigger a server side commit.
>   </description>
> </property>
>
>
> On Thu, Apr 25, 2013 at 4:35 PM, Bai Shen <ba...@gmail.com> wrote:
>
> > I'm having two problems with the solrindex job in Nutch 2.1
> >
> > When I run it with -all, it indexes every single parsed document, not
> just
> > the newly generated ones, as fetch and parse do.
> >
> > Secondly, it's adding my documents in small chunks.  I was fetching in
> 100
> > document cycles and when I run solrindex I get messages such as the
> > following.
> >
> > Adding 87 documents
> > Adding 5 documents
> > Adding 2 documents
> > Adding 3 documents
> > Adding 14 documents
> > Adding 34 documents
> > Adding 233 documents
> >
> > Any ideas what causes this and how to fix it?
> >
> > Thanks.
> >
>