IndexPartitioning

[Otis Gospodnetic <ot...@yahoo.com>]

Just came across http://wiki.apache.org/solr/IndexPartitioning .  Does anyone think we still need this idea (vs. MultiCore)?


Otis 
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

Re: IndexPartitioning

[Norberto Meijome <fr...@meijome.net>]

On Wed, 27 Aug 2008 16:29:40 -0700 (PDT)
Chris Hostetter <ho...@fucit.org> wrote:

> : interesting - it sounds to me more like shards than multicore, doesn't
> it? ... : but instead of querying all shards for results, only the shards
> that are
> 
> it's somewhat an amalgam of both ... there's the "index" aspect and the 
> "query" aspect ... you can imagine sending data to one Solr master and 
> then it decides based on some criteria which "partition" it belongs in ... 
> that partition might be a local "core" or it might be a remote "shard" ... 
> the key being that once the data has made it into a single partition 
> 9somewhere) queries that only care about a single partition could be made 
> directly to it, while queries that care about *all* the data would need to 
> hit all of the partitions (which in today's parlance would all be shards, 
> even if they are in fact just alternate cores on the same port)

Gotcha, thanks Chris. Yes, definitely like what PostgreSQL  does with
partitions (though I'm not sure pgsql supports partitions in remote servers! )

B
_________________________
{Beto|Norberto|Numard} Meijome

"The knowledge that makes us cherish
innocence makes innocence unattainable."
   Irving Howe

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

Re: IndexPartitioning

[Chris Hostetter <ho...@fucit.org>]

: interesting - it sounds to me more like shards than multicore, doesn't it? ...
: but instead of querying all shards for results, only the shards that are

it's somewhat an amalgam of both ... there's the "index" aspect and the 
"query" aspect ... you can imagine sending data to one Solr master and 
then it decides based on some criteria which "partition" it belongs in ... 
that partition might be a local "core" or it might be a remote "shard" ... 
the key being that once the data has made it into a single partition 
9somewhere) queries that only care about a single partition could be made 
directly to it, while queries that care about *all* the data would need to 
hit all of the partitions (which in today's parlance would all be shards, 
even if they are in fact just alternate cores on the same port)




-Hoss

Re: IndexPartitioning

[Norberto Meijome <fr...@meijome.net>]

On Mon, 25 Aug 2008 09:34:28 -0700 (PDT)
Otis Gospodnetic <ot...@yahoo.com> wrote:

> Just came across http://wiki.apache.org/solr/IndexPartitioning .  Does anyone think we still need this idea (vs. MultiCore)?
> 

interesting - it sounds to me more like shards than multicore, doesn't it? ...
but instead of querying all shards for results, only the shards that are
expected to contain the data we are after are queried. Similar to PostgreSQL's
(and Oracle's + others?) partitions by query. 

FWIW, I think it'd be very nice to handle this @ the request handler level,
rather than @ the data import/query stage .... Obviously, as an alternative
option to the current shards approach, for those cases where we can't (or don't
want to)  define the "partitions".

cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome

If you're not part of the solution, you're part of the precipitate.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.