Re: SolrCloud on Trunk

[Mark Miller <ma...@gmail.com>]

On Feb 28, 2012, at 9:33 AM, Jamie Johnson wrote:

> where specifically this is on the roadmap for SolrCloud.  Anyone
> else have those details?

I think we would like to do this sometime in the near future, but I don't know exactly what time frame fits in yet. There is a lot to do still, and we also need to get a 4 release of both Lucene and Solr out to users soon. It could be in a point release later - but it's open source - it really just depends on what people start doing it and get it done. I will say it's something I'd like to see done.

With what we have now, one option we have talked about in the past was to just install multiple shards on a single machine - later you can start up a replica on a new machine when you are ready to grow and kill the original shard.

i.e. you could startup 15 shards on a single machine, and then over time migrate shards off nodes and onto new hardware. It's as simple as starting up a new replica on the new hardware and removing the core on machines you want to stop serving that shard from. This would let you expand to a 15 shard/machine cluster with N replicas (scaling replicas is as simple as starting a new node or stopping an old one).

- Mark Miller
lucidimagination.com

Re: SolrCloud on Trunk

[Yonik Seeley <yo...@lucidimagination.com>]

On Thu, Mar 1, 2012 at 12:27 AM, Jamie Johnson <je...@gmail.com> wrote:
> Is there a ticket around doing this?

Around splitting shards?

The easiest thing to consider is just splitting a single shard in two
reusing some of the existing buffering/replication mechanisms we have.
1) create two new shards to represent each half of the old index
2) make sure leaders are forwarding udpates to them and that the
shards are buffering them
3) do a commit and split the current index
4) proceed with recovery as normal on the two new shards (replicate
the halfs, apply the buffered updates)
5) some unresolved stuff such as how to transition leadership from the
single big shard to the smaller shards.  maybe just handle like leader
failure.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

Re: SolrCloud on Trunk

[Jamie Johnson <je...@gmail.com>]

Mark,

Is there a ticket around doing this?  If the work/design was written
down somewhere the community might have a better idea of how exactly
we could help.

On Wed, Feb 29, 2012 at 11:21 PM, Mark Miller <ma...@gmail.com> wrote:
>
> On Feb 28, 2012, at 9:33 AM, Jamie Johnson wrote:
>
>> where specifically this is on the roadmap for SolrCloud.  Anyone
>> else have those details?
>
> I think we would like to do this sometime in the near future, but I don't know exactly what time frame fits in yet. There is a lot to do still, and we also need to get a 4 release of both Lucene and Solr out to users soon. It could be in a point release later - but it's open source - it really just depends on what people start doing it and get it done. I will say it's something I'd like to see done.
>
> With what we have now, one option we have talked about in the past was to just install multiple shards on a single machine - later you can start up a replica on a new machine when you are ready to grow and kill the original shard.
>
> i.e. you could startup 15 shards on a single machine, and then over time migrate shards off nodes and onto new hardware. It's as simple as starting up a new replica on the new hardware and removing the core on machines you want to stop serving that shard from. This would let you expand to a 15 shard/machine cluster with N replicas (scaling replicas is as simple as starting a new node or stopping an old one).
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>