Removing last replica from a SolrCloud collection

["David Smiley (@MITRE.org)" <DS...@mitre.org>]

Hi,

If I issue either a core UNLOAD command, or a collection DELETEREPLICA
command,  (which both seem pretty much equivalent) it works but if there are
no other replicas for the shard, then the metadata for the shard is
completely gone in clusterstate.json!  That's pretty disconcerting because
you're basically hosed.  Of course, why would I even want to do that?  Well
I'm experimenting with ways to restore a backed-up replica to replace
existing data for the shard.

If this is unexpected behavior then I'll file a bug.

~ David



-----
 Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Removing last replica from a SolrCloud collection

["David Smiley (@MITRE.org)" <DS...@mitre.org>]

Thanks; that is *exactly* the behavior I'm talking about.  I tried to find an
existing issue before posting but missed this one somehow.


Ramkumar R. Aiyengar wrote
> There's already an issue for this,
> https://issues.apache.org/jira/browse/SOLR-5209, we were once bitten by
> the
> same issue, when we were trying to relocate a shard. As Mark mentions, the
> idea was to do this in zk truth mode, the link also references where that
> work is being done.





-----
 Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772p4114973.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Removing last replica from a SolrCloud collection

["Ramkumar R. Aiyengar" <an...@gmail.com>]

There's already an issue for this,
https://issues.apache.org/jira/browse/SOLR-5209, we were once bitten by the
same issue, when we were trying to relocate a shard. As Mark mentions, the
idea was to do this in zk truth mode, the link also references where that
work is being done.
On 31 Jan 2014 23:10, "David Smiley (@MITRE.org)" <DS...@mitre.org> wrote:

> Hi,
>
> If I issue either a core UNLOAD command, or a collection DELETEREPLICA
> command,  (which both seem pretty much equivalent) it works but if there
> are
> no other replicas for the shard, then the metadata for the shard is
> completely gone in clusterstate.json!  That's pretty disconcerting because
> you're basically hosed.  Of course, why would I even want to do that?  Well
> I'm experimenting with ways to restore a backed-up replica to replace
> existing data for the shard.
>
> If this is unexpected behavior then I'll file a bug.
>
> ~ David
>
>
>
> -----
>  Author:
> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Removing last replica from a SolrCloud collection

[Mark Miller <ma...@gmail.com>]

We also really need to update the docs to use the collections API rather
than the pre configured core as well. I've been aiming to get the
preconfigured core (and hence collection) out of there by default for
solrcloud, but it's not easy, because I do think it still makes some sense
for non solrcloud mode and having another solr.xml to maintain and have the
user choose from kind of sucks.

One interesting idea is to simply leave it there and let the
zk=cluster_truth stuff take care of it. If you start in SolrCloud mode,
that core will belong to a collection that does not exist and be removed.

-- 
- Mark

http://about.me/markrmiller

On Sun, Feb 2, 2014 at 12:01 AM, Mark Miller <ma...@gmail.com> wrote:

> It's expected behaviour. It's expected to change with the zk=cluster_truth
> mode that I hope we can start squeezing into 4.7.
>
> In the first couple releases of SolrCloud, there was no collections API. I
> did not have time to make it. Even what went in for the collections api was
> banged out in a very short amount of spare time I had and still needs lot's
> of additional work.
>
> Anyway, with no collections API, each replica being unloaded and
> unregistered was the only way to delete a collection. From what I've read,
> the hash ranges in zk work did not take this into account. I have not
> looked closely into it myself, so I don't know if that should be considered
> a bug or a rock and a hard place.
>
> We are left with a few of these kinds of warts due to time frames and
> working in harmony with a lot of existing Solr stuff.
>
> A large majority of those issues will be easily resolved with the
> zk=cluster_truth mode. The reason we didn't have that mode before is that
> it requires not only a collections api, but a nice, full, solid collections
> API. Most of that I've done in my spare time, though some of the later work
> was sponsored by Cloudera.
>
> We have come a long way with the collections api, and shalin and noble are
> helping work done this path, so it doesn't seem far off.
>
> Some of the zk=cluster_truth stuff might take a bit of time to get, but
> the initial mode (which should end up the default, and the old mode likely
> deprecated) should be fairly easy to get going.
>
> If the issue is really so nasty, perhaps we break back compat around this,
> given that the collections api can handle it properly now. My feeling has
> been more, let's just get an initial zk=cluster_truth done - there are lot
> of warts without it, and an initial zk=cluster_truth is not too much work,
> which is why I easily think it could be in 4.7.
>
> I guess it depends - if we don't think users will switch to the new mode
> and want to do things around unloading every replica in a collection while
> keeping the collection around, perhaps we should break back compat around
> this behaviour and force use of the collections API. I'd file a JIRA issue
> if you feel strongly about it. I have not really had to deal with any
> fallout from that yet.
>
> What do you mean you are hosed?
>
> All the SolrCores should be gone, so why does it matter so much that the
> collection is gone? In the days before the collections api, if you wanted
> to then create the collection again, just create a solrcore, same way a
> collection was created initially.
>
> Without the zk=cluster_truth mode, we have to support both collections api
> collections, and pre configured implicit collections, created and removed
> via SolrCores.
>
> --
> - Mark
>
> http://about.me/markrmiller
>
> On Fri, Jan 31, 2014 at 6:08 PM, David Smiley (@MITRE.org) <
> DSMILEY@mitre.org> wrote:
>
>> Hi,
>>
>> If I issue either a core UNLOAD command, or a collection DELETEREPLICA
>> command,  (which both seem pretty much equivalent) it works but if there
>> are
>> no other replicas for the shard, then the metadata for the shard is
>> completely gone in clusterstate.json!  That's pretty disconcerting because
>> you're basically hosed.  Of course, why would I even want to do that?
>>  Well
>> I'm experimenting with ways to restore a backed-up replica to replace
>> existing data for the shard.
>>
>> If this is unexpected behavior then I'll file a bug.
>>
>> ~ David
>>
>>
>>
>> -----
>>  Author:
>> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
>
>

Re: Removing last replica from a SolrCloud collection

["David Smiley (@MITRE.org)" <DS...@mitre.org>]

Mark, I appreciate all the context.

...

Mark Miller-3 wrote
> What do you mean you are hosed?
> 
> All the SolrCores should be gone, so why does it matter so much that the
> collection is gone? In the days before the collections api, if you wanted
> to then create the collection again, just create a solrcore, same way a
> collection was created initially.
> 
> Without the zk=cluster_truth mode, we have to support both collections api
> collections, and pre configured implicit collections, created and removed
> via SolrCores.

You're hosed most especially because clusterstate.json loses knowledge of
what the hash range is for the shard (and everything else) once there are no
more replicas for the shard.  I'm not trying to delete the collection, I'm
trying to *replace* the replicas of the shards of the collection.  My
experiment was to unload the replicas (of just one shard; the others would
be handled later) but before I could get to adding a backed-up replica I
observed clusterstate.json lost all knowledge of what that shard was and
then I realize I was hosed.

But any way, this simply isn't going to work for the reasons you gave.  I'm
hoping for some way to clone or reconstitute a collection from a backup.  I
can manually create the collection similar to how it was created the first
time, but after shard-splitting I fear that isn't feasible since the backups
won't align with the new collection's shard hash ranges unless I can specify
what those hash-ranges are (but I can't).  Maybe manual clusterstate.json
surgery is inevitable.  This is what I raised in the other email I sent at a
similar time to the one your replied to.

~ David



-----
 Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772p4114905.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Removing last replica from a SolrCloud collection

[Mark Miller <ma...@gmail.com>]

It's expected behaviour. It's expected to change with the zk=cluster_truth
mode that I hope we can start squeezing into 4.7.

In the first couple releases of SolrCloud, there was no collections API. I
did not have time to make it. Even what went in for the collections api was
banged out in a very short amount of spare time I had and still needs lot's
of additional work.

Anyway, with no collections API, each replica being unloaded and
unregistered was the only way to delete a collection. From what I've read,
the hash ranges in zk work did not take this into account. I have not
looked closely into it myself, so I don't know if that should be considered
a bug or a rock and a hard place.

We are left with a few of these kinds of warts due to time frames and
working in harmony with a lot of existing Solr stuff.

A large majority of those issues will be easily resolved with the
zk=cluster_truth mode. The reason we didn't have that mode before is that
it requires not only a collections api, but a nice, full, solid collections
API. Most of that I've done in my spare time, though some of the later work
was sponsored by Cloudera.

We have come a long way with the collections api, and shalin and noble are
helping work done this path, so it doesn't seem far off.

Some of the zk=cluster_truth stuff might take a bit of time to get, but the
initial mode (which should end up the default, and the old mode likely
deprecated) should be fairly easy to get going.

If the issue is really so nasty, perhaps we break back compat around this,
given that the collections api can handle it properly now. My feeling has
been more, let's just get an initial zk=cluster_truth done - there are lot
of warts without it, and an initial zk=cluster_truth is not too much work,
which is why I easily think it could be in 4.7.

I guess it depends - if we don't think users will switch to the new mode
and want to do things around unloading every replica in a collection while
keeping the collection around, perhaps we should break back compat around
this behaviour and force use of the collections API. I'd file a JIRA issue
if you feel strongly about it. I have not really had to deal with any
fallout from that yet.

What do you mean you are hosed?

All the SolrCores should be gone, so why does it matter so much that the
collection is gone? In the days before the collections api, if you wanted
to then create the collection again, just create a solrcore, same way a
collection was created initially.

Without the zk=cluster_truth mode, we have to support both collections api
collections, and pre configured implicit collections, created and removed
via SolrCores.

-- 
- Mark

http://about.me/markrmiller

On Fri, Jan 31, 2014 at 6:08 PM, David Smiley (@MITRE.org) <
DSMILEY@mitre.org> wrote:

> Hi,
>
> If I issue either a core UNLOAD command, or a collection DELETEREPLICA
> command,  (which both seem pretty much equivalent) it works but if there
> are
> no other replicas for the shard, then the metadata for the shard is
> completely gone in clusterstate.json!  That's pretty disconcerting because
> you're basically hosed.  Of course, why would I even want to do that?  Well
> I'm experimenting with ways to restore a backed-up replica to replace
> existing data for the shard.
>
> If this is unexpected behavior then I'll file a bug.
>
> ~ David
>
>
>
> -----
>  Author:
> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Removing-last-replica-from-a-SolrCloud-collection-tp4114772.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>