You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Troy Edwards <te...@gmail.com> on 2016/02/08 16:15:13 UTC
Data Import Handler - autoSoftCommit and autoCommit
We are running the data import handler to retrieve about 10 million records
during work hours every day of the week. We are using Clean = true, Commit
= true and Optimize = true. The entire process takes about 1 hour.
What would be a good setting for autoCommit and autoSoftCommit?
Thanks
Re: Data Import Handler - autoSoftCommit and autoCommit
Posted by Susheel Kumar <su...@gmail.com>.
You can start with one of the suggestions from this link based on your
indexing and query load.
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Susheel
On Mon, Feb 8, 2016 at 10:15 AM, Troy Edwards <te...@gmail.com>
wrote:
> We are running the data import handler to retrieve about 10 million records
> during work hours every day of the week. We are using Clean = true, Commit
> = true and Optimize = true. The entire process takes about 1 hour.
>
> What would be a good setting for autoCommit and autoSoftCommit?
>
> Thanks
>
Re: Data Import Handler - autoSoftCommit and autoCommit
Posted by Rajesh Hazari <ra...@gmail.com>.
we have this for a collection which updated every 3mins with min of 500
documents and once in a day of 10k documents in start of the day
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:300000}</maxTime>
<maxDocs>10000</maxDocs>
<openSearcher>true</openSearcher>
<expungeDeletes>true</expungeDeletes>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:6000}</maxTime>
</autoSoftCommit>
As per solr documentation, If you have solr client to index documents,
its not suggested to use commit=true and optimize=true explicitly.
we have not tested data import handle with 10 million records.
we have settled with this config after many tests and after understanding
the need and requirements.
*Rajesh**.*
On Mon, Feb 8, 2016 at 10:15 AM, Troy Edwards <te...@gmail.com>
wrote:
> We are running the data import handler to retrieve about 10 million records
> during work hours every day of the week. We are using Clean = true, Commit
> = true and Optimize = true. The entire process takes about 1 hour.
>
> What would be a good setting for autoCommit and autoSoftCommit?
>
> Thanks
>