SolrCloud architecture overview doc?

[Ilan Ginzburg <il...@gmail.com>]

Hi,

I'm looking for a blog post or documentation giving a good overview of the
architecture of SolrCloud (not the user experience and use of Solr). The
audience would be engineers knowledgeable about distributed systems that
know nothing about SolrCloud (side note: I looked for something similar for
Elasticsearch and didn't find either).

Documentation that will touch on the index structure
(collection/shard/replica), the way they're materialized as cores, the
coordination done by ZooKeeper (shard leader élections). How these replicas
interact with each other and the update log in steady state and in recovery
scenarios.
The Overseer role (I wrote the detailed overseer doc
https://github.com/apache/solr/blob/main/dev-docs/overseer/overseer.adoc)
and how it interacts with ZooKeeper, the storage of all metadata in
ZooKeeper, the way nodes start, the way cluster state is propagated and
managed (watches for collections with replicas on a node vs all other
collection) etc.

If anybody knows about a doc that gives a complete overview, I'm very
interested.
Otherwise might end up writing it 🤓

Thanks,
Ilan

Re: SolrCloud architecture overview doc?

[Mark Miller <ma...@gmail.com>]

Maybe.

Yeah the song is just meant to be funny.

The long / extremely short, limited value, overview of my view is that
pretty much every classical way you can share clusterstate with Zookeeper
is an Overseer design, at least how I defined or thought of an Overseer
before it had any implementation . The fundamental concept of the Overseer
has always been some kind of cluster lock for some duration. If you
simplify a bit, I see how you implement an Overseer as a spectrum, where on
one end you have something like a Hadoop NameNode, or a permanent cluster
lock. Maybe you have a hot backup in the wings. As you move toward the
other end of the spectrum in this simplified model, you have cluster locks
that are held for less and less time, and maybe even more cluster locks
than one for some tasks. At the very far end, you have something like an
optimistic concurrency cluster lock, and effectively, ridiculously short
lock holds.

Personally, I would have aimed for the optimistic concurrency end of the
spectrum in implementation, but short of it, that is actually where I
started as a foothold, with no intention of staying there. But anyway, I
would have moved up the spectrum a short ways. I don’t think anyone really
aimed for where the current Overseer did land, it kind of gets there
through sheer weight and silliness as opposed to any grand design, but it’s
obviously spent most of its life closer to the NameNode end, while trying
to pretend it’s more in the middle. It very much could be where I initially
would have imagined it ended up with  extremely minimal functional changes
- I mean, 100% code change, but functional extremely close. Which isn’t to
say that is the ideal target. But one should understand that reality if
they want to understand what that monster actually is.

Going back to the Overseer looking much closer to a NameNode but kind of
pretending it’s more in the middle, Fundamentally, that’s indicative of
everything that’s wrong with the Overseer. It doesn’t know what it is. It
leans on Zookeeper like an unsure underseer.  And in the end, that helps
ensure it would never be good even if you wanted to target its location on
the spectrum as your desired location.

And of course that’s compounded by the ridiculous cluster communication
cadence it allows for, the more than ridiculous communication cost it
requires. And the hairy, complexity that fairly weak mitigation attempts
have added over time to the already hairy, way overly complex code.

And already, I can’t wrap that up in any meaningful way here, I’m already
way past my attempted word limit.

Re: SolrCloud architecture overview doc?

[Ilan Ginzburg <il...@gmail.com>]

Mark, can you please share the rest of the internal presentation or its
general idea?
Curious how much (or if) your take on Overseer differs from the
"distributed mode" option.

Granted, "distributed mode" does not rely on optimistic concurrency (which
I get from the lyrics of the song) but on distributed locking (no Overseer
song for that, sorry).
Optimistic concurrency (if that's implemented as distributed conditional
updates of state in ZK) would likely be a better option for low load
clusters (few races) but might not be so efficient on high load clusters
(many collisions).


On Fri, May 31, 2024 at 10:02 AM Mark Miller <ma...@gmail.com> wrote:

> I would disavow any of those videos as a technical resource.
>
> Funny enough, I was just wrapping up a small internal presentation to share
> my take on the Overseer. The writing has easily been eclipsed by this song
> I wrote for the outro though.
>
> https://youtu.be/VlJUJan9WGU
>

Re: SolrCloud architecture overview doc?

[Mark Miller <ma...@gmail.com>]

I would disavow any of those videos as a technical resource.

Funny enough, I was just wrapping up a small internal presentation to share
my take on the Overseer. The writing has easily been eclipsed by this song
I wrote for the outro though.

https://youtu.be/VlJUJan9WGU

Re: SolrCloud architecture overview doc?

[Mark Miller <ma...@gmail.com>]

I would disavow any of those videos as a technical resource.

Funny enough, I was just wrapping up a small internal presentation to share
my take on the Overseer. The writing has easily been eclipsed by this song
I wrote for the outro though.

https://youtu.be/VlJUJan9WGU

Re: SolrCloud architecture overview doc?

[Ilan Ginzburg <il...@gmail.com>]

Thanks Mike (and Mark), I didn't know of these videos.
Still looking for a broader (and TBH deeper) coverage of the
architecture (including changes from the last 11 years :) ideally in
text/diagram form so if anybody has something else, please share.

Ilan


On Thu, May 30, 2024 at 1:33 PM Mike Drob <md...@mdrob.com> wrote:
>
> I think Mark has given this as a presentation a few times, maybe
> https://www.youtube.com/watch?v=eVK0wLkLw9w
>
> On Thu, May 30, 2024 at 3:03 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm looking for a blog post or documentation giving a good overview of the
> > architecture of SolrCloud (not the user experience and use of Solr). The
> > audience would be engineers knowledgeable about distributed systems that
> > know nothing about SolrCloud (side note: I looked for something similar for
> > Elasticsearch and didn't find either).
> >
> > Documentation that will touch on the index structure
> > (collection/shard/replica), the way they're materialized as cores, the
> > coordination done by ZooKeeper (shard leader élections). How these replicas
> > interact with each other and the update log in steady state and in recovery
> > scenarios.
> > The Overseer role (I wrote the detailed overseer doc
> > https://github.com/apache/solr/blob/main/dev-docs/overseer/overseer.adoc)
> > and how it interacts with ZooKeeper, the storage of all metadata in
> > ZooKeeper, the way nodes start, the way cluster state is propagated and
> > managed (watches for collections with replicas on a node vs all other
> > collection) etc.
> >
> > If anybody knows about a doc that gives a complete overview, I'm very
> > interested.
> > Otherwise might end up writing it 🤓
> >
> > Thanks,
> > Ilan
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org

Re: SolrCloud architecture overview doc?

[Mike Drob <md...@mdrob.com>]

I think Mark has given this as a presentation a few times, maybe
https://www.youtube.com/watch?v=eVK0wLkLw9w

On Thu, May 30, 2024 at 3:03 AM Ilan Ginzburg <il...@gmail.com> wrote:

> Hi,
>
> I'm looking for a blog post or documentation giving a good overview of the
> architecture of SolrCloud (not the user experience and use of Solr). The
> audience would be engineers knowledgeable about distributed systems that
> know nothing about SolrCloud (side note: I looked for something similar for
> Elasticsearch and didn't find either).
>
> Documentation that will touch on the index structure
> (collection/shard/replica), the way they're materialized as cores, the
> coordination done by ZooKeeper (shard leader élections). How these replicas
> interact with each other and the update log in steady state and in recovery
> scenarios.
> The Overseer role (I wrote the detailed overseer doc
> https://github.com/apache/solr/blob/main/dev-docs/overseer/overseer.adoc)
> and how it interacts with ZooKeeper, the storage of all metadata in
> ZooKeeper, the way nodes start, the way cluster state is propagated and
> managed (watches for collections with replicas on a node vs all other
> collection) etc.
>
> If anybody knows about a doc that gives a complete overview, I'm very
> interested.
> Otherwise might end up writing it 🤓
>
> Thanks,
> Ilan
>