You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Troy Edwards <te...@gmail.com> on 2016/02/08 16:15:13 UTC

Data Import Handler - autoSoftCommit and autoCommit

We are running the data import handler to retrieve about 10 million records
during work hours every day of the week. We are using Clean = true, Commit
= true and Optimize = true. The entire process takes about 1 hour.

What would be a good setting for autoCommit and autoSoftCommit?

Thanks

Re: Data Import Handler - autoSoftCommit and autoCommit

Posted by Susheel Kumar <su...@gmail.com>.
You can start with one of the suggestions from this link based on your
indexing and query load.


https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/


Thanks,
Susheel

On Mon, Feb 8, 2016 at 10:15 AM, Troy Edwards <te...@gmail.com>
wrote:

> We are running the data import handler to retrieve about 10 million records
> during work hours every day of the week. We are using Clean = true, Commit
> = true and Optimize = true. The entire process takes about 1 hour.
>
> What would be a good setting for autoCommit and autoSoftCommit?
>
> Thanks
>

Re: Data Import Handler - autoSoftCommit and autoCommit

Posted by Rajesh Hazari <ra...@gmail.com>.
we have this for a collection which updated every 3mins with min of 500
documents and once in a day of 10k documents in start of the day

<autoCommit>
       <maxTime>${solr.autoCommit.maxTime:300000}</maxTime>
<maxDocs>10000</maxDocs>
        <openSearcher>true</openSearcher>
        <expungeDeletes>true</expungeDeletes>
     </autoCommit>
   <autoSoftCommit>
      <maxTime>${solr.autoSoftCommit.maxTime:6000}</maxTime>
   </autoSoftCommit>

As per solr documentation, If you have solr client to index documents,
its not suggested to use commit=true and optimize=true explicitly.

we have not tested data import handle with 10 million records.

we have settled with this config after many tests and after understanding
the need and requirements.


*Rajesh**.*

On Mon, Feb 8, 2016 at 10:15 AM, Troy Edwards <te...@gmail.com>
wrote:

> We are running the data import handler to retrieve about 10 million records
> during work hours every day of the week. We are using Clean = true, Commit
> = true and Optimize = true. The entire process takes about 1 hour.
>
> What would be a good setting for autoCommit and autoSoftCommit?
>
> Thanks
>