ping query frequency

[adm1n <ev...@gmail.com>]

Hi,


I'm wonderring how frequent this query should be made. Currently it is done
before each select request (some very old legacy). I googled a little and
found out that it is bad practice and has performance impact. So the
question is should I completely remove it or just do it once in some period
of time.

What is the best practice?


thanks



--
View this message in context: http://lucene.472066.n3.nabble.com/ping-query-frequency-tp4044305.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ping query frequency

[Shawn Heisey <so...@elyograg.org>]

On 3/4/2013 2:06 AM, adm1n wrote:
> here you go - http://wiki.solarium-project.org/index.php/V1:Ping_query
>
>
> as for my cloud, average response time for ping request is 4 ms. but there
> are several pings that take even 3 seconds. (I have about 100000 pings/day)

I would suspect GC pauses, like I was having.  The following startup 
options are what I use now, and I am no longer having really long GC 
pauses.  They are specifically designed to not require any specific -Xmx 
value.

-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 
-XX:NewRatio=3 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled 
-XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts

A note: I was using some of these options already, even when the pauses 
(sometimes up to 12 seconds!) were happening.  It was when I added these 
particular options that the problem went away:

-XX:CMSInitiatingOccupancyFraction=75 -XX:NewRatio=3 
-XX:MaxTenuringThreshold=8 -XX:+ParallelRefProcEnabled -XX:+UseLargePages

I later added AggressiveOpts, which means "for all options that will get 
new defaults in the next version of java, change to those new defaults." 
  This option has been known to cause problems with some programs, but 
so far I have not seen any problems with Solr 3.5 or 4.2-SNAPSHOT.

Thanks for the Solarium link.  I think their advice is good - don't do 
it before every request.  For me, doing it every five seconds probably 
allows a few dozen queries to go by between each ping.

A question to the experts: Do any of the various Jenkins processes use 
AggressiveOpts?  Would I use the same commandline option to turn it on 
when I run the tests myself with ant, or is there a different syntax?

Thanks,
Shawn

Re: ping query frequency

[adm1n <ev...@gmail.com>]

*Shawn:*
here you go - http://wiki.solarium-project.org/index.php/V1:Ping_query


as for my cloud, average response time for ping request is 4 ms. but there
are several pings that take even 3 seconds. (I have about 100000 pings/day)



--
View this message in context: http://lucene.472066.n3.nabble.com/ping-query-frequency-tp4044305p4044472.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ping query frequency

[Amit Nithian <an...@gmail.com>]

We too run a ping every 5 seconds and I think the concurrent Mark/Sweep
helps to avoid the LB from taking a box out of rotation due to long pauses.
Either that or I don't see large enough pauses for my LB to take it out
(it'd have to fail 3 times in a row or 15 seconds total before it's gone).

The ping query does execute an actual query so of course you want to make
this as simple as possible (i.e. q=<primary_key>:<value>) so that there's
limited to no scanning of the index. I think our query does an id:0 which
would always return 0 docs but also any stupid-simple query is fine so long
as it hits the caches on subsequent hits. The goal, to me at least, is not
that the ping query yields actual docs but that it's a mechanism to remove
a solr server out of rotation without having to login to an "ops
controlled" device directly.

I'd definitely remove the ping per request (wouldn't the fact that you are
doing /select serve as the "ping" and hence defeat the purpose of the ping
query) and definitely do the frequent ping as we are describing if you want
to have your solr boxes behind some load balancer.


On Sun, Mar 3, 2013 at 8:21 AM, Shawn Heisey <so...@elyograg.org> wrote:

> On 3/3/2013 2:15 AM, adm1n wrote:
>
>> I'm wonderring how frequent this query should be made. Currently it is
>> done
>> before each select request (some very old legacy). I googled a little and
>> found out that it is bad practice and has performance impact. So the
>> question is should I completely remove it or just do it once in some
>> period
>> of time.
>>
>
> Can you point me at the place where it says that it's bad practice to do
> frequent pings?  I use the ping functionality in my haproxy load balancer
> that sits in front of Solr.  It executes a ping request against all my Solr
> instances every five seconds.  Most of the time, the ping request (which is
> distributed) finishes in single-digit milliseconds. If that is considered
> bad practice, I want to figure out why and submit issues to get the problem
> fixed.
>
> I can imagine that sending a ping before every query would be a bad idea,
> but I am hoping that the way I'm using it is OK.
>
> The only problem with ping requests that I have ever noticed was caused by
> long garbage collection pauses on my 8GB Solr heap.  Those pauses caused
> the load balancer to incorrectly mark the active Solr instance(s) as down
> and send requests to a backup.
>
> Through experimentation with -XX memory tuning options, I have now
> eliminated the GC pause problem.  For machines running Solr 4.2-SNAPSHOT, I
> have reduced the heap to 6GB, the 3.5.0 machines are still running with 8GB.
>
> Thanks,
> Shawn
>
>

Re: ping query frequency

[Shawn Heisey <so...@elyograg.org>]

On 3/3/2013 2:15 AM, adm1n wrote:
> I'm wonderring how frequent this query should be made. Currently it is done
> before each select request (some very old legacy). I googled a little and
> found out that it is bad practice and has performance impact. So the
> question is should I completely remove it or just do it once in some period
> of time.

Can you point me at the place where it says that it's bad practice to do 
frequent pings?  I use the ping functionality in my haproxy load 
balancer that sits in front of Solr.  It executes a ping request against 
all my Solr instances every five seconds.  Most of the time, the ping 
request (which is distributed) finishes in single-digit milliseconds. If 
that is considered bad practice, I want to figure out why and submit 
issues to get the problem fixed.

I can imagine that sending a ping before every query would be a bad 
idea, but I am hoping that the way I'm using it is OK.

The only problem with ping requests that I have ever noticed was caused 
by long garbage collection pauses on my 8GB Solr heap.  Those pauses 
caused the load balancer to incorrectly mark the active Solr instance(s) 
as down and send requests to a backup.

Through experimentation with -XX memory tuning options, I have now 
eliminated the GC pause problem.  For machines running Solr 
4.2-SNAPSHOT, I have reduced the heap to 6GB, the 3.5.0 machines are 
still running with 8GB.

Thanks,
Shawn