You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2007/04/27 19:50:40 UTC
Solr and memcached
Hi,
I'm considering adding support for caching results in memcached. Questions:
1. Has anyone already done this? (searched, didn't find anything)
2. Can anyone think of any reasons why this might not be a good idea? (I *just* started considering this)
3. I read http://wiki.apache.org/solr/SolrCaching , but I think the whole cache discarding and warming wouldn't be needed if what I store in memcached is: StringRepresentationOfQuery > DocSet or DocList . Hm, I see QueryResultKey class now. Then maybe I'd store QueryResultKey -> DocSet or DocList in the memcached cache. Is this a correct?
Thanks,
Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/ - Tag - Search - Share
Re: Solr and memcached
Posted by James liu <li...@gmail.com>.
i used to think cache data with memcached.
why i think that?
for example, i have 45 solr instance, and i have to merge their results into
on array and sort by score or datetime, if i use rows=10, it means i will
get max 45*10 results,,but it only show 10result per page.
how to do with 440 results,,just kill them or save them into memcached?
i saved them into memcached.
it seems only save merge and sort time because i find same query just
pagenum changed the response will be very quick.
Merge and sort time is no much when i test. And if use memcached, we have to
manage it ,add hardware and have to learn how to use it, solve its problem.
So now i not use memcached and just use solr *inside* cache.
2007/4/28, Chris Hostetter <ho...@fucit.org>:
>
>
> : 2. Can anyone think of any reasons why this might not be a good idea? (I
> : *just* started considering this)
>
> : 3. I read http://wiki.apache.org/solr/SolrCaching , but I think the
> : whole cache discarding and warming wouldn't be needed if what I store in
> : memcached is: StringRepresentationOfQuery > DocSet or DocList . Hm, I
> : see QueryResultKey class now. Then maybe I'd store QueryResultKey ->
> : DocSet or DocList in the memcached cache. Is this a correct?
>
> The nice thing about the internal caching is that because it's internal,
> it can be autowarmed, and it can store things that only make sense as part
> of the internal API (ie: big OpenBitSet based DocSets that rely on the
> IndexReader for getting the real field contents)
>
> when you start talking about caching the data outside of Solr, I don't
> think the "internal" SolrCache APIs make as much sense anymore, what you
> can effectively/efficiently cache changes, and it may not make sense to
> hook that cache in at such a low level anymore -- it starts making more
> sense to talk about caching request=>response pairs (with
> pagination and field lists baked into them) instead of caching
> individual DocLists and DocSets ... at that level hooking into memcached
> might make sense, but it's probably easier and just as effective to use
> something like squid as a proxy cache in front of SOlr instead.
>
> (that's what i do)
>
>
>
> -Hoss
>
>
--
regards
jl
Re: Solr and memcached
Posted by Chris Hostetter <ho...@fucit.org>.
: 2. Can anyone think of any reasons why this might not be a good idea? (I
: *just* started considering this)
: 3. I read http://wiki.apache.org/solr/SolrCaching , but I think the
: whole cache discarding and warming wouldn't be needed if what I store in
: memcached is: StringRepresentationOfQuery > DocSet or DocList . Hm, I
: see QueryResultKey class now. Then maybe I'd store QueryResultKey ->
: DocSet or DocList in the memcached cache. Is this a correct?
The nice thing about the internal caching is that because it's internal,
it can be autowarmed, and it can store things that only make sense as part
of the internal API (ie: big OpenBitSet based DocSets that rely on the
IndexReader for getting the real field contents)
when you start talking about caching the data outside of Solr, I don't
think the "internal" SolrCache APIs make as much sense anymore, what you
can effectively/efficiently cache changes, and it may not make sense to
hook that cache in at such a low level anymore -- it starts making more
sense to talk about caching request=>response pairs (with
pagination and field lists baked into them) instead of caching
individual DocLists and DocSets ... at that level hooking into memcached
might make sense, but it's probably easier and just as effective to use
something like squid as a proxy cache in front of SOlr instead.
(that's what i do)
-Hoss
Re: Solr and memcached
Posted by Ken Krugler <kk...@transpac.com>.
Hi Otis,
>I'm considering adding support for caching results in memcached. Questions:
>
>1. Has anyone already done this? (searched, didn't find anything)
Not exactly, but we do something similar to this for Nutch searches
using ehcache (http://krugle.com/kse/projects/eFNJEmX). But we store
the (rewritten) query string and then the serialized XML response, as
that way we don't have dependencies on stable searcher/doc ids (which
is, for Nutch, the only reference we get from the remote searchers).
So depending on the # of entries in your cache and the size of the
hit (docs * XML representation for each), storing the XML might be a
reasonable option.
-- Ken
>2. Can anyone think of any reasons why this might not be a good
>idea? (I *just* started considering this)
>3. I read http://wiki.apache.org/solr/SolrCaching , but I think the
>whole cache discarding and warming wouldn't be needed if what I
>store in memcached is: StringRepresentationOfQuery > DocSet or
>DocList . Hm, I see QueryResultKey class now. Then maybe I'd store
>QueryResultKey -> DocSet or DocList in the memcached cache. Is this
>a correct?
>
>Thanks,
>Otis
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Re: Solr and memcached
Posted by Yonik Seeley <yo...@apache.org>.
If you store internal docids, then you need to add the specific reader
(or index version?) as part of the key since the ids are transient.
-Yonik
On 4/27/07, Otis Gospodnetic <ot...@yahoo.com> wrote:
> Hi,
>
> I'm considering adding support for caching results in memcached. Questions:
>
> 1. Has anyone already done this? (searched, didn't find anything)
> 2. Can anyone think of any reasons why this might not be a good idea? (I *just* started considering this)
> 3. I read http://wiki.apache.org/solr/SolrCaching , but I think the whole cache discarding and warming wouldn't be needed if what I store in memcached is: StringRepresentationOfQuery > DocSet or DocList . Hm, I see QueryResultKey class now. Then maybe I'd store QueryResultKey -> DocSet or DocList in the memcached cache. Is this a correct?
>
> Thanks,
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/ - Tag - Search - Share