How to limit the number of open searchers?

[Michael Ryan <mr...@moreover.com>]

Is there a way to limit the number of searchers that can be open at a given time?  I know there is a maxWarmingSearchers configuration that limits the number of warming searchers, but that's not quite what I'm looking for...

Ideally, when I commit, I want there to only be one searcher open before the commit, so that during the commit and warming, there is a max of two searchers open.  I'd be okay with delaying the commit until there is only one searcher open.  Is there a way to programmatically determine how many searchers are currently open?

-Michael

RE: How to limit the number of open searchers?

[Michael Ryan <mr...@moreover.com>]

> I'm curious, why can't you do a master/slave setup?

It's just not all that useful for this particular application. Indexing new docs and merging segments - which as I understand is the main strength of having a write-only master - is a relatively small part of our app. What really is expensive perf-wise for us is the facet regeneration after each commit, and long-running complex queries. I'm not sure a master/slave setup would solve the issue with many open searchers - even with a long poll interval, won't the slave still potentially open many searchers if old searchers are still in use?

My current plan is to create a request handler that iterates through the mbeans in req.getCore().getInfoRegistry() and gets a count of how many distinct SolrIndexSearchers there are, then returns this count. Seems really hacky, but fairly straightforward. Would love to hear any ideas on a better way to do this.

-Michael

Re: How to limit the number of open searchers?

[Erick Erickson <er...@gmail.com>]

Ah, you're right. If you queries run across several commits
you'll get multiple searchers open.

I don't know of any good way to do what you want.

I'm curious, why can't you do a master/slave setup?

The other thing to think about would be the NRT stuff
if you can run trunk.

Best
Erick

On Wed, Mar 7, 2012 at 2:30 PM, Michael Ryan <mr...@moreover.com> wrote:
>> Unless you have warming happening, there should
>> only be a single searcher open at any given time.
>> So it seems to me that maxWarmingSearchers
>> should give you what you need.
>
> What I'm seeing is that if a query takes a very long time to run, and runs across the duration of multiple commits (I know, that itself sounds bad!), I can get into a situation where I have 2 searchers in use and 1 searcher warming, rather than 1 searcher in use and 1 searcher warming. Due to all the memory-intensive features I use, having 3 or more searchers open can cause an OutOfMemoryError.
>
> I'm not using master/slave for this application, so can't go that route.
>
> I'd like a way to see how many searchers are currently open that is external to Solr. This would allow me to block my commits until I see that there is only 1 searcher currently open. I could use JMX, but that feels like overkill - wondering if there is something simpler.
>
> -Michael

RE: How to limit the number of open searchers?

[Michael Ryan <mr...@moreover.com>]

> Unless you have warming happening, there should
> only be a single searcher open at any given time.
> So it seems to me that maxWarmingSearchers
> should give you what you need.

What I'm seeing is that if a query takes a very long time to run, and runs across the duration of multiple commits (I know, that itself sounds bad!), I can get into a situation where I have 2 searchers in use and 1 searcher warming, rather than 1 searcher in use and 1 searcher warming. Due to all the memory-intensive features I use, having 3 or more searchers open can cause an OutOfMemoryError.

I'm not using master/slave for this application, so can't go that route.

I'd like a way to see how many searchers are currently open that is external to Solr. This would allow me to block my commits until I see that there is only 1 searcher currently open. I could use JMX, but that feels like overkill - wondering if there is something simpler.

-Michael

Re: How to limit the number of open searchers?

[Erick Erickson <er...@gmail.com>]

Unless you have warming happening, there should
only be a single searcher open at any given time.
So it seems to me that maxWarmingSearchers
should give you what you need.

And you can pretty easily insure this by making your
poll interval (assuming master/slave) longer
than your warmup time.

Best
Erick

On Mon, Mar 5, 2012 at 2:18 PM, Michael Ryan <mr...@moreover.com> wrote:
> Is there a way to limit the number of searchers that can be open at a given time?  I know there is a maxWarmingSearchers configuration that limits the number of warming searchers, but that's not quite what I'm looking for...
>
> Ideally, when I commit, I want there to only be one searcher open before the commit, so that during the commit and warming, there is a max of two searchers open.  I'd be okay with delaying the commit until there is only one searcher open.  Is there a way to programmatically determine how many searchers are currently open?
>
> -Michael

Re: How to limit the number of open searchers?

[Li Li <fa...@gmail.com>]

what do u mean "programmatically"? modify codes of solr? becuase solr is
not like lucene, it only provide http interfaces for its users other than
java api

if you want to modify solr, you can find codes in SolrCore
private final LinkedList<RefCounted<SolrIndexSearcher>> _searchers = new
LinkedList<RefCounted<SolrIndexSearcher>>();
and _searcher is current searcher.
be careful to use searcherLock to synchronizing your codes.
maybe you can write your codes like:

synchronized(searcherLock){
    if(_searchers.size==1){
        ...
    }
}




On Tue, Mar 6, 2012 at 3:18 AM, Michael Ryan <mr...@moreover.com> wrote:

> Is there a way to limit the number of searchers that can be open at a
> given time?  I know there is a maxWarmingSearchers configuration that
> limits the number of warming searchers, but that's not quite what I'm
> looking for...
>
> Ideally, when I commit, I want there to only be one searcher open before
> the commit, so that during the commit and warming, there is a max of two
> searchers open.  I'd be okay with delaying the commit until there is only
> one searcher open.  Is there a way to programmatically determine how many
> searchers are currently open?
>
> -Michael
>