You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Frank Wesemann <f....@fotofinder.net> on 2011/07/15 19:15:43 UTC

DIH full-import - when is commit() actally triggered?

Hello,
I am running a full import with a quite plain data-config (a root entity 
with three sub entities ) from a jdbc datasource.
This import is expected to add approximately 10 mio documents
What I now see from my logfiles is, that a newSearcher event is fired 
about every five seconds.
This causes a lot load on the machine.
While searching "*:*" via the admin interface it appears, that on every 
new commit about 1.000 docs are newly added.
This the "batchSize" I configured in the datasource definition, but I 
don't think that this related.
in solrconfig I have

<updateHandler class="solr.DirectUpdateHandler2" enable="true">
    <maxPendingDeletes>100000</maxPendingDeletes>
    <autoCommit>
            <maxDocs>100000</maxDocs>  <!-- maximum uncommited docs before autocommit triggered -->
            <maxTime>300000</maxTime>
     </autoCommit>
</updateHandler>


What other parameters in solrconfig.xml or in my data-config may be 
related to this behaviour?
Any hint is appreciated.

Thanks
frank

-- 
mit freundlichem Gruß,

Frank Wesemann
Fotofinder GmbH         USt-IdNr. DE812854514
Software Entwicklung    Web: http://www.fotofinder.com/
Potsdamer Str. 96       Tel: +49 30 25 79 28 90
10785 Berlin            Fax: +49 30 25 79 28 999

Sitz: Berlin
Amtsgericht Berlin Charlottenburg (HRB 73099)
Geschäftsführer: Ali Paczensky

Re: DIH full-import - when is commit() actally triggered?

Posted by Frank Wesemann <f....@fotofinder.net>.
Ahmet Arslan schrieb:
>> I am running a full import with a quite plain data-config
>> (a root entity with three sub entities ) from a jdbc
>> datasource.
>> This import is expected to add approximately 10 mio
>> documents
>> What I now see from my logfiles is, that a newSearcher
>> event is fired about every five seconds.
>>     
>
> This is triggered by autoCommit in every 300,000 milli seconds.
> You need to remove <maxTime>300000</maxTime> to disable this mechanism.
>
>
>   
Thanks Ahmet,
indeed I had to remove the <maxDocs> Entry. So now a commit happens only 
every five minutes.

-- 
mit freundlichem Gruß,

Frank Wesemann
Fotofinder GmbH         USt-IdNr. DE812854514
Software Entwicklung    Web: http://www.fotofinder.com/
Potsdamer Str. 96       Tel: +49 30 25 79 28 90
10785 Berlin            Fax: +49 30 25 79 28 999

Sitz: Berlin
Amtsgericht Berlin Charlottenburg (HRB 73099)
Geschäftsführer: Ali Paczensky

Re: DIH full-import - when is commit() actally triggered?

Posted by Ahmet Arslan <io...@yahoo.com>.
> I am running a full import with a quite plain data-config
> (a root entity with three sub entities ) from a jdbc
> datasource.
> This import is expected to add approximately 10 mio
> documents
> What I now see from my logfiles is, that a newSearcher
> event is fired about every five seconds.

This is triggered by autoCommit in every 300,000 milli seconds.
You need to remove <maxTime>300000</maxTime> to disable this mechanism.