SolrJ + HTTP caching

[Jon Baer <jo...@gmail.com>]

Hi,

What is the proper behavior suppose to be between SolrJ and caching?   
Im proxying through a framework and wondering if it is possible to  
turn on / turn off caching programatically depending on the type of  
query (or if this will have no effect whatsoever) ... since SolrJ uses  
Apache HTTP client libs can it negotiate anything here?

SOLR-127: HTTP Caching awareness.  Solr now recognizes HTTP Request
     headers related to HTTP Caching (see RFC 2616 sec13) and will  
respond
     with "304 Not Modified" when appropriate.  New options have been  
added
     to solrconfig.xml to influence this behavior.
     (Thomas Peuss via hossman)

Thanks.

- Jon

Re: SolrJ + HTTP caching

[Norberto Meijome <nu...@gmail.com>]

On Wed, 15 Oct 2008 11:11:07 -0700
Matthew Runo <mr...@zappos.com> wrote:

> We've been using Varnish (http://varnish.projects.linpro.no/) in front  
> of our Solr servers, and have been seeing about a 70% hit rate for the  
> queries. We're using SolrJ, and have seen no bad effects of the cache.

FWIW : 
We also use Varnish in front of SOLR - we refresh the index daily, so we have a
fairly long TTL, but clear it at the end of the script which calls DIH.
The web app also caches rendered results (webpages :P)  in memcached.

B

_________________________
{Beto|Norberto|Numard} Meijome

"Build a system that even a fool can use, and only a fool will want to use it."
   George Bernard Shaw

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

Re: SolrJ + HTTP caching

[Chris Hostetter <ho...@fucit.org>]

: with the index being a few minutes stale as the TTL expires on the cache. I
: don't think solr has a way to, at query time, change the cache control
: headers.

SolrJ lets the HttpClient instance handle all network connections, so 
specify whatever caching/proxy info you want to it, and then pass it to 
your CommonsHttpSolrServer constructor.

you should be able to easily a new CommonsHttpSolrServer for each request 
(they're pretty cheap to construct) or have N CommonsHttpSolrServer each 
with a different HttpClient instance for supporting N different 
caching/proxying prefrences (just pick which CommonsHttpSolrServer to use 
on each request depending on what behavior you want)



-Hoss

Re: SolrJ + HTTP caching

[Matthew Runo <mr...@zappos.com>]

We've been using Varnish (http://varnish.projects.linpro.no/) in front  
of our Solr servers, and have been seeing about a 70% hit rate for the  
queries. We're using SolrJ, and have seen no bad effects of the cache.

That said, we're just caching everything for a few minutes. We don't  
pick and choose which queries get cached in Varnish, and our business  
users are fine with the index being a few minutes stale as the TTL  
expires on the cache. I don't think solr has a way to, at query time,  
change the cache control headers.

http://wiki.apache.org/solr/SolrAndHTTPCaches may be a good jumping  
off point for more thought.

Thanks for your time!

Matthew Runo
Software Engineer, Zappos.com
mruno@zappos.com - 702-943-7833

On Oct 15, 2008, at 9:24 AM, Jon Baer wrote:

> Hi,
>
> What is the proper behavior suppose to be between SolrJ and  
> caching?  Im proxying through a framework and wondering if it is  
> possible to turn on / turn off caching programatically depending on  
> the type of query (or if this will have no effect whatsoever) ...  
> since SolrJ uses Apache HTTP client libs can it negotiate anything  
> here?
>
> SOLR-127: HTTP Caching awareness.  Solr now recognizes HTTP Request
>    headers related to HTTP Caching (see RFC 2616 sec13) and will  
> respond
>    with "304 Not Modified" when appropriate.  New options have been  
> added
>    to solrconfig.xml to influence this behavior.
>    (Thomas Peuss via hossman)
>
> Thanks.
>
> - Jon
>