commit / new searcher delay?

[Steve Conover <sc...@gmail.com>]

I'm doing some testing of a solr master/slave config and find that,
after syncing my slave, I need to sleep for about 400ms after commit
to "see" the new index state.  i.e. if I don't sleep, and I execute a
query, I get results that reflect the prior state of the index.

I suspect this has something to do with waiting for the searcher to
warm and switch over (?).  Though, I'm confused because when I print
out /solr/admin/registry.jsp, the hashcode of the Searcher changes
immediately (as the commit docs say, the commit operation blocks by
default until a new searcher is in place).  I've tried turning off all
caching, to no effect.

Anyone have any idea what could be going on here?  Ideally, <commit/>
would be an operation that blocks until the exact moment when the new
searcher is in place and is actually serving based on the new index
state...

-Steve

Re: commit / new searcher delay?

[sunnyfr <jo...@gmail.com>]

Hi Hossman,

I would love to know either how do you manage this ? 

thanks,


Shalin Shekhar Mangar wrote:
> 
> On Fri, Mar 6, 2009 at 8:47 AM, Steve Conover <sc...@gmail.com> wrote:
> 
>> That's exactly what I'm doing, but I'm explicitly replicating, and
>> committing.  Even under these circumstances, what could explain the
>> delay after commit before the new index becomes available?
>>
> 
> How are you explicitly replicating? I mean, how do you make sure that the
> slave has actually finished replication and the new index is available
> now?
> Are you using the script based replication or the new java based one?
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: http://www.nabble.com/commit---new-searcher-delay--tp22342916p23036207.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: commit / new searcher delay?

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Fri, Mar 6, 2009 at 8:47 AM, Steve Conover <sc...@gmail.com> wrote:

> That's exactly what I'm doing, but I'm explicitly replicating, and
> committing.  Even under these circumstances, what could explain the
> delay after commit before the new index becomes available?
>

How are you explicitly replicating? I mean, how do you make sure that the
slave has actually finished replication and the new index is available now?
Are you using the script based replication or the new java based one?

-- 
Regards,
Shalin Shekhar Mangar.

Re: commit / new searcher delay?

[Steve Conover <sc...@gmail.com>]

That's exactly what I'm doing, but I'm explicitly replicating, and
committing.  Even under these circumstances, what could explain the
delay after commit before the new index becomes available?

On Thu, Mar 5, 2009 at 10:55 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Thu, Mar 5, 2009 at 10:30 PM, Steve Conover <sc...@gmail.com> wrote:
>
>> Yep, I notice the default is true/true, but I explicitly specified
>> both those things too and there's no difference in behavior.
>>
>
> Perhaps you are indexing on the master and then searching on the slaves? It
> may be the delay introduced by replication.
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: commit / new searcher delay?

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Thu, Mar 5, 2009 at 10:30 PM, Steve Conover <sc...@gmail.com> wrote:

> Yep, I notice the default is true/true, but I explicitly specified
> both those things too and there's no difference in behavior.
>

Perhaps you are indexing on the master and then searching on the slaves? It
may be the delay introduced by replication.
-- 
Regards,
Shalin Shekhar Mangar.

Re: commit / new searcher delay?

[Steve Conover <sc...@gmail.com>]

Yep, I notice the default is true/true, but I explicitly specified
both those things too and there's no difference in behavior.

On Wed, Mar 4, 2009 at 7:39 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Thu, Mar 5, 2009 at 6:06 AM, Steve Conover <sc...@gmail.com> wrote:
>
>> I'm doing some testing of a solr master/slave config and find that,
>> after syncing my slave, I need to sleep for about 400ms after commit
>> to "see" the new index state.  i.e. if I don't sleep, and I execute a
>> query, I get results that reflect the prior state of the index.
>>
>
> How are you sending the commit? You should use commit with waitSearcher=true
> and waitFlush=true so that it blocks until the new searcher becomes
> available for querying.
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: commit / new searcher delay?

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Thu, Mar 5, 2009 at 6:06 AM, Steve Conover <sc...@gmail.com> wrote:

> I'm doing some testing of a solr master/slave config and find that,
> after syncing my slave, I need to sleep for about 400ms after commit
> to "see" the new index state.  i.e. if I don't sleep, and I execute a
> query, I get results that reflect the prior state of the index.
>

How are you sending the commit? You should use commit with waitSearcher=true
and waitFlush=true so that it blocks until the new searcher becomes
available for querying.


-- 
Regards,
Shalin Shekhar Mangar.

Re: commit / new searcher delay?

[Chris Hostetter <ho...@fucit.org>]

: I suspect this has something to do with waiting for the searcher to
: warm and switch over (?).  Though, I'm confused because when I print
: out /solr/admin/registry.jsp, the hashcode of the Searcher changes
: immediately (as the commit docs say, the commit operation blocks by
: default until a new searcher is in place).  I've tried turning off all
: caching, to no effect.

off the top of my head i don't remember if registry.jsp will start to list 
hte new searcher even if it's not swapped in to become hte current 
searcher.

what you should check, is when the slave logs hte end of the commit, and 
how long that is after the start of the commit. (and of course: if it logs 
any warming stats -- you might have overlooked a cache)

: Anyone have any idea what could be going on here?  Ideally, <commit/>
: would be an operation that blocks until the exact moment when the new
: searcher is in place and is actually serving based on the new index

that's how it works if you use waitSearcher=true ... are you sure you're 
not seeing the effects of some caching (HTTP?) in between solr and your 
client?  did you try setting never304=true in the <httpCaching> section of 
solrconfig.xml ?



-Hoss