Indexing Halts for long time and then restarts

["Husain, Yavar" <yh...@firstam.com>]

Since Erick is really active answering now so posting a quick question :)

I am using:
DIH
Solr 3.5 on Windows

Building Auto Recommendation Utility

Having around 1 Billion Query Strings (3-6 words each) in database. Indexing them using NGram.

Merge Factor = 30
Auto Commit not set.

DIH halted after indexing 7 million for around 25 minutes and was not showing any increment in the Total Documents Processed/Fetched, ofcourse it was doing some stuff, was it some merge stuff?. After 25 minutes it started moving again.

Due to this indexing time has increased a lot. Any help will be appreciated.

Thanks.
</PRE>
<BR>
******************************************************************************************<BR>This message may contain confidential or proprietary information intended only for the use of the<BR>addressee(s) named above or may contain information that is legally privileged. If you are<BR>not the intended addressee, or the person responsible for delivering it to the intended addressee,<BR>you are hereby notified that reading, disseminating, distributing or copying this message is strictly<BR>prohibited. If you have received this message by mistake, please immediately notify us by<BR>replying to the message and delete the original message and any copies immediately thereafter.<BR>
<BR>
Thank you.~<BR>
******************************************************************************************<BR>
FAFLD<BR>
<PRE>

Re: Indexing Halts for long time and then restarts

[Erick Erickson <er...@gmail.com>]

Flattery will get you a lot <G>...

Yeah, I expect you're hitting a merge issue. To test, set up autocommit
to only trigger after a lot of docs are committed. You should see the
time before the big pause change radically (perhaps disappear if
you don't commit until the run is done).

Note that it'll still happen, just not as often. This problem is changed
in 4.0 with the DocumentWriterPerThread stuff (Mike McCandless
wrote a cool blog post on it).


Best
Erick

On Fri, Mar 16, 2012 at 8:27 AM, Husain, Yavar <yh...@firstam.com> wrote:
> Since Erick is really active answering now so posting a quick question :)
>
> I am using:
> DIH
> Solr 3.5 on Windows
>
> Building Auto Recommendation Utility
>
> Having around 1 Billion Query Strings (3-6 words each) in database. Indexing them using NGram.
>
> Merge Factor = 30
> Auto Commit not set.
>
> DIH halted after indexing 7 million for around 25 minutes and was not showing any increment in the Total Documents Processed/Fetched, ofcourse it was doing some stuff, was it some merge stuff?. After 25 minutes it started moving again.
>
> Due to this indexing time has increased a lot. Any help will be appreciated.
>
> Thanks.
> </PRE>
> <BR>
> ******************************************************************************************<BR>This message may contain confidential or proprietary information intended only for the use of the<BR>addressee(s) named above or may contain information that is legally privileged. If you are<BR>not the intended addressee, or the person responsible for delivering it to the intended addressee,<BR>you are hereby notified that reading, disseminating, distributing or copying this message is strictly<BR>prohibited. If you have received this message by mistake, please immediately notify us by<BR>replying to the message and delete the original message and any copies immediately thereafter.<BR>
> <BR>
> Thank you.~<BR>
> ******************************************************************************************<BR>
> FAFLD<BR>
> <PRE>