SOLR-1395 integration with katta. Question about Katta's ranking among shards and IDF's

[Marc Sturlese <ma...@gmail.com>]

Hey there,
I am trying to set up the Katta integration plugin. I would like to know if
Katta's ranking algorith is used when searching among shards. In case yes,
would it mean it solves the problem with IDF's of distributed Solr? 
-- 
View this message in context: http://www.nabble.com/SOLR-1395-integration-with-katta.-Question-about-Katta%27s-ranking-among-shards-and-IDF%27s-tp25819241p25819241.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR-1395 integration with katta. Question about Katta's ranking among shards and IDF's

[Jason Rutherglen <ja...@gmail.com>]

Marc,

What do you mean by Katta's ranking algorithm? If you use
SOLR-1395's search request system that traverses Hadoop RPC,
it's simply using what Solr offers today in terms of distributed
search (i.e. no distributed IDF). Instead of requests being
serialized into an HTTP call, they are serialized over Hadoop
RPC. That's the only difference. Katta's main benefit to me
isn't it's integration with search, but it's management of
shard/cores over a large cluster. This is a major benefit to
Solr, where today, if a server goes down or slows down (which is
worse), all queries start to back up and fail, and the shard
proxy server needs to be restarted because dead threads have
exceeded the app server's thread pool limit. This should not
happen on production servers. Also Solr needs the ability to
return partial results if a core is unavailable, this will be
much easier to implement with SOLR-1395 because the dead node
may be marked as such in Zookeeper and future requests sent to
an available replicated shard.

It should be fairly easy to add to Katta (if it's not there
already) the ability to send a request to the shard/core with
the lowest load. There's also recently been work in adding new
shard distribution policies.

-J

On Fri, Oct 9, 2009 at 4:03 AM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Hey there,
> I am trying to set up the Katta integration plugin. I would like to know if
> Katta's ranking algorith is used when searching among shards. In case yes,
> would it mean it solves the problem with IDF's of distributed Solr?
> --
> View this message in context: http://www.nabble.com/SOLR-1395-integration-with-katta.-Question-about-Katta%27s-ranking-among-shards-and-IDF%27s-tp25819241p25819241.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>