[jira] [Commented] (SOLR-4509) Disable Stale Check - Distributed Search (Performance)

["Ryan Zezeski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786290#comment-13786290 ] 

Ryan Zezeski commented on SOLR-4509:
------------------------------------

I recently updated Yokozuna (1) to use Solr 4.4.0.  After running a
query benchmark I noticed that throughput had dropped to 44% of the
baseline.  After some head scratching I realized that my distributed
search patch had not applied successfully.  Sure enough, after I
updated the patch for 4.4.0 throughput returned to 100%+ of baseline.
Below is a table showing results of query benchmark for 4.3.0, 4.4.0
and 4.4.0 without this patch.  The throughput drops to less than half
of Solr 4.4.0 with the patch and the latency more than doubles.

|Measurement     |Solr 4.3.0       |Solr 4.4.0       |Solr 4.4.0 w/o Patch  |
|----------------|-----------------|-----------------|----------------------|
|Mean Throughput |1512 ops/s       |1525 ops/s       |670 ops/s (44%)       |
|Median Latency  |22.0ms           |21.6ms           |46.2ms (2.1x)         |
|95th Latency    |29.8ms           |29.4ms           |76.8ms (2.6x)         |
|99th Latency    |35.3ms           |34.6ms           |86.2ms (2.5x)         |

These results are against a 4-node cluster all hosted on 1 physical
machine.  Manual distributed search is used for querying, there is no
use of SolrCloud.  There are only 1 million small text documents
stored.  The query matches only 1 of these documents.  The query
results and filter caches are enabled and should have a high hit
ratio.  The point is to make the queries inexpensive as possible to
see what other overhead might occur.  There may very well be scenarios
where this patch makes little to no difference.  But in this case it
seems to make a big one.

This update is not to prove that my patch makes a significant
difference in all cases.  Rather, I accidentally ran this benchmark
and was surprised at the difference I saw.  I wanted to ping this
issue in hopes that others might try the patch to see if it helps.

Here is the corresponding ticket on the Yokozuna repo: https://github.com/basho/yokozuna/pull/197

1: Yokozuna is a project with integrates Solr with the Riak database.  https://github.com/basho/yokozuna

> Disable Stale Check - Distributed Search (Performance)
> ------------------------------------------------------
>
>                 Key: SOLR-4509
>                 URL: https://issues.apache.org/jira/browse/SOLR-4509
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>         Environment: 5 node SmartOS cluster (all nodes living in same global zone - i.e. same physical machine)
>            Reporter: Ryan Zezeski
>            Priority: Minor
>         Attachments: baremetal-stale-nostale-med-latency.dat, baremetal-stale-nostale-med-latency.svg, baremetal-stale-nostale-throughput.dat, baremetal-stale-nostale-throughput.svg, IsStaleTime.java, SOLR-4509.patch
>
>
> By disabling the Apache HTTP Client stale check I've witnessed a 2-4x increase in throughput and reduction of over 100ms.  This patch was made in the context of a project I'm leading, called Yokozuna, which relies on distributed search.
> Here's the patch on Yokozuna: https://github.com/rzezeski/yokozuna/pull/26
> Here's a write-up I did on my findings: http://www.zinascii.com/2013/solr-distributed-search-and-the-stale-check.html
> I'm happy to answer any questions or make changes to the patch to make it acceptable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org