Adding shards dynamically when the automatic routing is used

[mganeshs <mg...@live.in>]

All,

Is there possiblity in near future in coming new releases, adding shards
dynamically though compositeId ( default ) based routing is used.
Currently only option is we need to split the shard, instead we should able
to add shards dynamically and then on all new documents should go on new
shards.
Is there a plan to include this in coming release(s) ?

Regards,
Ganesh



--
View this message in context: http://lucene.472066.n3.nabble.com/Adding-shards-dynamically-when-the-automatic-routing-is-used-tp4333883.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Adding shards dynamically when the automatic routing is used

[Shawn Heisey <ap...@elyograg.org>]

On 5/8/2017 2:07 AM, mganeshs wrote:
> Is there possiblity in near future in coming new releases, adding
> shards dynamically though compositeId ( default ) based routing is
> used. Currently only option is we need to split the shard, instead we
> should able to add shards dynamically and then on all new documents
> should go on new shards. Is there a plan to include this in coming
> release(s) ? 

Once a shard layout is created with compositeId routing and you index
data into that layout, you can't change it.  Each shard contains
documents that hash to a certain range of hash values.  If you change
the hash ranges without changing what documents are actually in each
shard (by reindexing) then SolrCloud functionality breaks, because the
internal consistency won't be there.

There is an issue to implement a rebalance API -- which could for
instance let you change from six shards to ten shards.  If implemented,
it involves completely rewriting the entire index, likely across
multiple servers, moving data from an old shard layout to a new one. 
The patch that has been supplied does apparently work, but it is
necessarily a relatively slow process.

https://issues.apache.org/jira/browse/SOLR-9241

Activity on the issue has stalled.  I do not know why.  It might be
because the patch includes no tests, so we are not assured that the
patch is as bulletproof as we want it to be.

Thanks,
Shawn