Re: Discuss SIP-14 Embedded Zookeeper

[Mike Drob <md...@apache.org>]

I put together a draft PR for the first stage of the SIP which is to cut
over unit tests to using the embedded ZK. There's a few things missing,
like setting limiters and our injected use of ZKDatabase, but we've also
had discussions about how those are not particularly useful as is.

The biggest change is the removal of SolrZooKeeper class, which was mainly
a reflection scaffolding around getting internals. We can use zookeeper
testing constructs a bit more cleanly for this, but is does affect the
public API for clients.

https://github.com/apache/solr/pull/522

On Wed, Sep 15, 2021 at 9:17 AM Mike Drob <md...@apache.org> wrote:

> > If people are running 100 or 1000 node clusters and use each node as a
> ZK server, by default, what kind of impact would that have?
>
> Bad. Very very bad. The largest ZK quorum I've personally seen is 9, and
> I've heard rumors of somebody running 15. I think the recommended approach
> for distributing load is to use Observers[1], which may provide some
> tiering benefits or may be redundant with traditional ZK clients. Maybe an
> Observer per failure zone makes sense for the Solr Operator?
>
> [1]: https://zookeeper.apache.org/doc/current/zookeeperObservers.html
>
>
>
> On Wed, Sep 15, 2021 at 8:33 AM Houston Putman <ho...@apache.org> wrote:
>
>> If we were to make this work, and support productionized embedded
>> Zookeeper, then it would absolutely be something that we want to support by
>> default in the Solr Operator.
>>
>> I don't think we'd be able to cut the Zookeeper Operator dependency
>> really quickly, because this is going in at the earliest in Solr 9 and more
>> likely Solr 10 (probably). The Solr Operator still needs to support older
>> versions, especially Solr 8 for a fair amount of time. So once the minimum
>> supported Solr version is one that has this feature, then we can get rid of
>> the Zookeeper Operator for good. This is probably my favorite thing about
>> the SIP. The Zookeeper Operator is fine, but removing that dependency would
>> lift a huge burden off of the Solr Operator's shoulders.
>>
>> I also think it's a good idea to be able to start solr in a ZK-Only mode.
>>
>> Also you should be able to tell Solr whether you want it to start as a ZK
>> member or observer, or not run ZK on that node at all. I'm not extremely in
>> touch with the ZK community at this point, but what cluster sizes are
>> people scaling up to nowadays? If people are running 100 or 1000 node
>> clusters and use each node as a ZK server, by default, what kind of impact
>> would that have?
>>
>> On Tue, Sep 14, 2021 at 8:20 PM Mike Drob <md...@apache.org> wrote:
>>
>>> I like the idea of starting nodes in a ZK-only mode, probably we would
>>> call it something like coordination mode. It ties in to ideas that I've had
>>> while discussing with other folks about other Solr node specializations,
>>> like "edge" nodes that are part of a cluster but do not host collections
>>> and exist solely for routing http queries to the appropriate places.
>>>
>>> I think it could be useful in a k8s deployment as well, but I'd have to
>>> think about how we want to do all the port magic there. I know that I've
>>> had conversations with Houston about wanting to move away from
>>> ZookeeperOperator, but those haven't quite taken hold yet.
>>>
>>> On Tue, Sep 14, 2021 at 6:02 PM Jan Høydahl <ja...@cominvent.com>
>>> wrote:
>>>
>>>> Thanks for kicking this off Mike. I added a few "rejected alternatives"
>>>> and put a few questions for thought in a comment. You may want to keep all
>>>> discussion in this email thread, so here are the questions copied:
>>>>
>>>>
>>>> *This is promising! Question: Would this mode be valuable also for
>>>> Kubernetes deployments, i.e. we could get rid of the ZookeeperOperator and
>>>> instead let the SolrOperator keep track of which Solr pods that also act as
>>>> ZK nodes?Would we allow a Solr node to start in a ZK-only mode, i.e. not
>>>> eligible for collections/cores/overseer? This would also support those huge
>>>> clusters where you want dedicated ZKs.*
>>>>
>>>> This also ties in with SIP-6 Solr should own the bootstrap process
>>>> <https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process>,
>>>> as we'd want to control startup/shutdown behavior wrt the zk, e.g. start
>>>> embedded zk before solr and stop solr before stopping zk. Perhaps also
>>>> gracefully exiting the quorum on planned shutdown?
>>>>
>>>> Jan
>>>>
>>>> 14. sep. 2021 kl. 22:09 skrev Mike Drob <md...@apache.org>:
>>>>
>>>> Devs,
>>>>
>>>> We've previously discussed maintaining ZK as being an operational
>>>> hurdle for some groups getting started or migrating to SolrCloud from
>>>> non-ZK cloud mode. I'd like to discuss the idea of embedding ZK in our own
>>>> process control.
>>>>
>>>> Please see the SIP at
>>>> https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper
>>>>
>>>> Thank you,
>>>> Mike
>>>>
>>>>
>>>>