commit is taking very very long time

[bbarani <bb...@gmail.com>]

Hi,

I am not sure why some commits take very long time. I have a batch indexing
which commits just once after it completes the indexing.

I tried to index just 36 rows but the total time taken to index was like 12
minutes. The indexing time was very less just some 30 seconds but it took
the remaining time for commit.


<response>
−
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
</lst>
−
<lst name="initArgs">
−
<lst name="defaults">
<str name="config">dataimportHydrogen.xml</str>
</lst>
</lst>
<str name="status">idle</str>
<str name="importResponse"/>
−
<lst name="statusMessages">
<str name="Total Requests made to DataSource">4</str>
<str name="Total Rows Fetched">36</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2010-07-22 15:42:28</str>
−
<str name="">
Indexing completed. Added/Updated: 4 documents. Deleted 0 documents.
</str>
<str name="Committed">2010-07-22 15:54:49</str>
<str name="Optimized">2010-07-22 15:54:49</str>
<str name="Total Documents Processed">4</str>
<str name="Time taken ">0:12:21.632</str>
</lst>
−
<str name="WARNING">
This response format is experimental.  It is likely to change in the future.
</str>
</response>


I even set the autowarm count to 0 in solrconfig.xml file but of non use.
Any reason why the commit takes more time?

Also is there a way to reduce the time it takes?

I have attached my solrconfig / log for your reference.

http://lucene.472066.n3.nabble.com/file/n988220/SOLRerror.log SOLRerror.log 
http://lucene.472066.n3.nabble.com/file/n988220/solrconfig.xml
solrconfig.xml 

Thanks,
BB


-- 
View this message in context: http://lucene.472066.n3.nabble.com/commit-is-taking-very-very-long-time-tp988220p988220.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: commit is taking very very long time

[Mark Miller <ma...@gmail.com>]

On 7/23/10 5:59 PM, Alexey Serba wrote:

> Another option is to set optimize=false in DIH call ( it's true by
> default ). 

Ouch - that should really be changed then.

- Mark

Re: commit is taking very very long time

[Alexey Serba <as...@gmail.com>]

> I am not sure why some commits take very long time.
Hmm... Because it merges index segments... How large is your index?

> Also is there a way to reduce the time it takes?
You can disable commit in DIH call and use autoCommit instead. It's
kind of hack because you postpone commit operation and make it async.

Another option is to set optimize=false in DIH call ( it's true by
default ). Also you can try to increase mergeFactor parameter but it
would affect search performance.