Re: Optimal Cache Settings, complicated by regular commits

[Chris Hostetter <ho...@fucit.org>]

: I'm trying to work out the optimum cache settings for our Solr server, I'll
: begin by outlining our usage.

...but you didn't give any information about what your cache settings look 
like ... size is only part of the picture, the autowarm counts are more 
significant.

: Commit frequency: sometimes we do massive amounts of sequential commits,

if you know you are going to be indexing more docs soon, then you can hold 
off on issuing a commit ... it really comes down to what kind of SLA you 
have to provide on how quickly an add/update is visible in the index -- 
don't commit any more often then that.

: The problem we have is that the default cache settings resulting in very low
: hit rates (less than 30% for documents, less than 1% for filterCache), so we

under 1% for filterCache sounds like you either have some really unique 
filter queries, or you are using enum based faceting on a huge field and 
the LRU cache is working against you by expunging values during a single 
request ... what version of solr are you using? what do the fieldtype 
declarations look like for the fields you are faceting on? what do the 
luke stats look like for hte fields you are faceting on?

: now we have the issue of commits being very slow (more than 5 seconds for a
: document), to the point where it causes a timeout elsewhere in our systems.
: This is made worse by the fact that committing seems to empty the cache,
: given that it takes about an hour to get the cache to a good state this is
: obviously very problematic.

1) using waitSearch=false can help speed up the commit if all you care 
about is not having your client time out.

2) using autowarming can help fill the caches up prior to users making 
requests (you may already know that, but since you didn't provide your 
cache configs i have no idea) .. they key is finding a good autowarm count 
that helps your cache stats w/o taking too long to fill up.


-Hoss