You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2025/08/20 22:35:26 UTC
Re: Propose: Disabled Overseer in Solr 10 as default
Update: I proposed the Solr version go into cluster properties[1] on ZK
initialization but Houston pushed back on that approach. The other
approach is to rely on the "least Solr version" as returned by the Solr
version being added to live nodes[2]. However that's very dynamic and I'm
concerned about an old Solr version somehow joining an existing cluster.
Rather than simply tell users "don't do that" (which we should do anyway),
I'm inclined to have Solr fail to join an existing cluster if doing so
would *lower* the least Solr version. Perhaps ignoring the final patch
version. I plan on updating that PR accordingly. The PR would have to go
into 9.x too, and thus such logic can't be enforced for older Solr
versions. Regardless, an upgrading user can choose the setting.
[1] https://issues.apache.org/jira/browse/SOLR-17664
[2] https://issues.apache.org/jira/browse/SOLR-17620
On Tue, Jul 22, 2025 at 1:12 AM David Smiley <ds...@apache.org> wrote:
> On the epic of seeking the _eventual_ demise of the Overseer, I'm seeking
> to make it disabled for *new* SolrCloud clusters in Solr 10. --
> https://issues.apache.org/jira/browse/SOLR-17293
> The epic: https://issues.apache.org/jira/browse/SOLR-14927 (oddly no SIP
> but it has a doc anyway). I think it's sufficiently ready for the great
> majority of SolrCloud clusters. A cluster with a collection containing a
> thousand+ replicas might pose a performance concern on start/restart events
> due to independent replica state updates. Of course, with such a change,
> there will be a section in the upgrade page in the ref guide to advise
> users who may opt to make an explicit choice.
>
> I don't love that the new mode doesn't have an elegant/clear way to refer
> to it. The best I've come up with is to say what it *isn't* -- it *isn't*
> the Overseer. "The Overseer is disabled". Awkwardly there are two
> undocumented solr.xml booleans, both a mouthful:
> distributedClusterStateUpdates and
> distributedCollectionConfigSetExecution. I propose instead that a single
> boolean cluster property be defined named "overseer" or "overseerEnabled".
> FYI the existing known cluster properties are defined here:
> org.apache.solr.common.cloud.ZkStateReader#KNOWN_CLUSTER_PROPS
> Even if such a boolean is agreeable... it raises the question of what
> should become of the "overseer" node role.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Anshum Gupta <an...@anshumgupta.net>.
I agree, I'm worried about the same i.e. defaulting to a mode that comes
with caveats and isn't really tested and hardened.
On Wed, Oct 1, 2025 at 9:35 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > Let's imagine (quite realistically) that I'm updating our upgrade notes
> and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
>
> Even assuming we can figure out a value on 'N' that sounds reasonable,
> can be shown to be stable in load testing, etc....is that enough to
> recommend "non-overseer" mode?
>
> Sizing (which includes replica-count) is notoriously a guess-and-check
> process in Solr. And even for users who have done everything right
> and dialed in their replica-count with some benchmarking - what
> happens when their requirements change and they need to add replicas
> to (e.g.) accommodate a higher QPS. Is there an easy way for those
> users to switch back to the overseer, or do they have to risk
> instability going forward?
>
> I guess I'm worried about basing recommendations on a factor like
> replica-count which has a tendency to drift over time, if the decision
> itself (i.e. overseer or not) is difficult to reverse after the fact.
> Not 100% sure that's the case here, but that's my suspicion based on a
> hazy recollection of some past discussions.
>
> Best,
>
> Jason
>
> On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > It's hard to provide a recommended threshold on collection size for
> > distributed mode.
> > I didn't run tests and it obviously depends on the number of nodes in the
> > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > that below a couple hundred replicas total for a collection it should be
> ok.
> > When a Solr node starts it marks all its replicas DOWN before marking
> them
> > ACTIVE. If PRS is not used this could take a long time with distributed
> > mode and be slower than Overseer due to the lack of batching of updates.
> >
> > Indexing and query performance is obviously not impacted by distributed
> > mode or Overseer performance, unless shard split performance is
> considered
> > part of indexing performance.
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> > >
> > > Sufficient stability is the characteristic I'm looking for in the above
> > > question, not performance. I understand that it'd be ideal if the
> > > cluster's state was structured differently to improve performance of
> > > certain administrative operations, but performance is not a
> requirement for
> > > stability. The most important performance considerations our users
> have
> > > relate to index & query. There's a basic assumption that nodes can
> restart
> > > in a "reasonable time"... maybe you'd care to try to define that. I
> think
> > > your recommendations around restructuring the cluster state would
> > > ultimately impact the performance of restarts and some other
> administrative
> > > scenarios but shouldn't be a prerequisite for a stable system.
> > >
> > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > > Distributed mode doesn't behave nicely when there are many concurrent
> > > > updates to a given collection's state.json.
> > > >
> > > > I'd recommend *against* making it the default at this time.
> > > >
> > > > The "root cause" is the presence of replica specific information in
> > > > state.json. In addition to relatively rare cases of changes to the
> > > sharding
> > > > of the collection, state.json is updated when replicas are created or
> > > > destroyed or moved or have their properties changed, and when PRS is
> not
> > > > used, also when replicas change state (which happens a lot when a
> Solr
> > > node
> > > > restarts for example).
> > > >
> > > > Therefore before making distributed mode the default, something has
> to be
> > > > done.
> > > > As Pierre suggests, redesign Collection API operations that require
> > > > multiple updates to be more efficient and group them when executing
> in
> > > > distributed mode. Also make sure that smaller operations that happen
> > > > concurrently are efficient enough.
> > > > Another option is to remove replica information from state.json (keep
> > > > collection metadata and shard definitions there), and create state-
> > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > Contention on anything replica related will be restricted to
> replicas of
> > > > the same shard.
> > > > There will be more watches on ZooKeeper, they will trigger less
> often and
> > > > less data will be read each time. Also less data to
> compress/uncompress
> > > > each time state.json is written or read (when so configured).
> > > >
> > > > Throttling goes against making SolrCloud as fast as we can.
> > > >
> > > > SolrCloud started with a single clusterstate.json file describing all
> > > > collections (removed in 9.0), then moved to per collection state.json
> > > files
> > > > for scalability reasons.
> > > > Maybe the time has come to split that big blob further?
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > hossman_lucene@fucit.org
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > : I don't think this should prevent shipping a system that is
> > > objectively
> > > > > way
> > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> matter
> > > > what
> > > > > : the default is. Changing the default makes it easier to remove
> it in
> > > > > Solr
> > > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > > amazing!
> > > > >
> > > > > I'm not understanding yoru claim that changing a default from A(x)
> to
> > > > A(y)
> > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > >
> > > > > You could change the default in 10.1, 10.2, etc... and it would
> still
> > > be
> > > > > the same amount of effort to remove it in 11.0.
> > > > >
> > > > > No matter when you change the default, if the *option* to use A(x)
> > > still
> > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > implementing
> > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> some
> > > > > code/process/documentation enabling users to migrate their cluster
> to
> > > > > A(y)
> > > > >
> > > > >
> > > > > -Hoss
> > > > > http://www.lucidworks.com/
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
Anshum Gupta
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by David Smiley <ds...@apache.org>.
Anshum, it's been "tested" since February 2021, so we're talking 4.5 years
now:
https://github.com/apache/solr/commit/23755ddfdd36a9613010cb9e6201127df55be744#diff-e49e335c95e0b12e7beda444cf67542457800de47dae39034af46a6797327776R125
(half of most SolrCloud test runs).
"Hardening" (say real world users, not tests): There is a chicken & egg
problem here, as it's opt-in. I'm glad Salesforce *finally* (after having
developed the thing! Jeesh) flipped the switch and shared some feedback
pertinent to large collections, albeit not pertinent to small or medium
ones (IMO). For many users, the Overseer is effectively an invisible
internal detail, so flipping this isn't going to be on people's radar. It
certainly wasn't in the past because the feature was extremely invisible /
undocumented. I recently (frantically?) worked on addressing that concern,
and giving it a name (overseer disabled). Even if users never care about
this (and why should they TBH), as a maintainer of SolrCloud, I care about
this very much! And really, you all should too. I can't think of any
positive improvement to SolrCloud's *maintainability* with a bigger impact
than removing the Overseer. Removing it is a gift to ourselves, and to any
user/operator of a cluster who has the misfortune of debugging Overseer
interactions.
I was pondering keeping both modes active in a cluster on a per-collection
basis (like PRS can do) which doesn't sound hard... but perhaps any time
spent on that would be better spent on improving the instability. If there
was a performance test designed to expose the instability, it'd do wonders
to prove out relatively cheap work-arounds like I suggested. Re-designing
cluster state, on the other hand, is a noble but costly endeavor. We'd
still need a test _anyway_ to know if the instability is fixed, or a brave
user :-)
On Wed, Oct 1, 2025 at 7:03 PM Ilan Ginzburg <il...@gmail.com> wrote:
> The design doc:
>
> https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit
>
> Timothy Potter identifying the issue right away:
>
> https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit?disco=AAAAHGxplOw
>
>
>
>
> On Thu, Oct 2, 2025 at 12:09 AM Ishan Chattopadhyaya
> <ic...@gmail.com> wrote:
> >
> > Is there a SIP or design document for this feature (no overseer mode),
> > please?
> > I'm familiar with the traditional overseer based mode, the PRS mode, not
> > very familiar with the DSU thing that Ilan + colleagues introduced. But,
> I
> > can't find enough design notes for the no overseer mode.
> >
> > On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
> >
> > > Sounds like an experimental feature that could end up changing (to fix
> this
> > > issue with replica count). While we want feedback from real users, this
> > > discussion makes me think it's just not ready for default status yet.
> > >
> > > On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <gerlowskija@gmail.com
> >
> > > wrote:
> > >
> > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes
> > > > and
> > > > > providing advice for users to choose the Overseer or not. Given
> the
> > > > > benefits & risks, at what replica count threshold (for the biggest
> > > > > collection) would you advise continued use of the Overseer?
> > > >
> > > > Even assuming we can figure out a value on 'N' that sounds
> reasonable,
> > > > can be shown to be stable in load testing, etc....is that enough to
> > > > recommend "non-overseer" mode?
> > > >
> > > > Sizing (which includes replica-count) is notoriously a
> guess-and-check
> > > > process in Solr. And even for users who have done everything right
> > > > and dialed in their replica-count with some benchmarking - what
> > > > happens when their requirements change and they need to add replicas
> > > > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > > > users to switch back to the overseer, or do they have to risk
> > > > instability going forward?
> > > >
> > > > I guess I'm worried about basing recommendations on a factor like
> > > > replica-count which has a tendency to drift over time, if the
> decision
> > > > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > > > Not 100% sure that's the case here, but that's my suspicion based on
> a
> > > > hazy recollection of some past discussions.
> > > >
> > > > Best,
> > > >
> > > > Jason
> > > >
> > > > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> > > wrote:
> > > > >
> > > > > It's hard to provide a recommended threshold on collection size for
> > > > > distributed mode.
> > > > > I didn't run tests and it obviously depends on the number of nodes
> in
> > > the
> > > > > cluster and how fast everything (including ZooKeeper) runs, but
> I'd say
> > > > > that below a couple hundred replicas total for a collection it
> should
> > > be
> > > > ok.
> > > > > When a Solr node starts it marks all its replicas DOWN before
> marking
> > > > them
> > > > > ACTIVE. If PRS is not used this could take a long time with
> distributed
> > > > > mode and be slower than Overseer due to the lack of batching of
> > > updates.
> > > > >
> > > > > Indexing and query performance is obviously not impacted by
> distributed
> > > > > mode or Overseer performance, unless shard split performance is
> > > > considered
> > > > > part of indexing performance.
> > > > >
> > > > > Ilan
> > > > >
> > > > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > > > notes and
> > > > > > providing advice for users to choose the Overseer or not. Given
> the
> > > > > > benefits & risks, at what replica count threshold (for the
> biggest
> > > > > > collection) would you advise continued use of the Overseer?
> > > > > >
> > > > > > Sufficient stability is the characteristic I'm looking for in the
> > > above
> > > > > > question, not performance. I understand that it'd be ideal if
> the
> > > > > > cluster's state was structured differently to improve
> performance of
> > > > > > certain administrative operations, but performance is not a
> > > > requirement for
> > > > > > stability. The most important performance considerations our
> users
> > > > have
> > > > > > relate to index & query. There's a basic assumption that nodes
> can
> > > > restart
> > > > > > in a "reasonable time"... maybe you'd care to try to define
> that. I
> > > > think
> > > > > > your recommendations around restructuring the cluster state would
> > > > > > ultimately impact the performance of restarts and some other
> > > > administrative
> > > > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > > > >
> > > > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <
> ilansolr@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Distributed mode doesn't behave nicely when there are many
> > > concurrent
> > > > > > > updates to a given collection's state.json.
> > > > > > >
> > > > > > > I'd recommend *against* making it the default at this time.
> > > > > > >
> > > > > > > The "root cause" is the presence of replica specific
> information in
> > > > > > > state.json. In addition to relatively rare cases of changes to
> the
> > > > > > sharding
> > > > > > > of the collection, state.json is updated when replicas are
> created
> > > or
> > > > > > > destroyed or moved or have their properties changed, and when
> PRS
> > > is
> > > > not
> > > > > > > used, also when replicas change state (which happens a lot
> when a
> > > > Solr
> > > > > > node
> > > > > > > restarts for example).
> > > > > > >
> > > > > > > Therefore before making distributed mode the default,
> something has
> > > > to be
> > > > > > > done.
> > > > > > > As Pierre suggests, redesign Collection API operations that
> require
> > > > > > > multiple updates to be more efficient and group them when
> executing
> > > > in
> > > > > > > distributed mode. Also make sure that smaller operations that
> > > happen
> > > > > > > concurrently are efficient enough.
> > > > > > > Another option is to remove replica information from state.json
> > > (keep
> > > > > > > collection metadata and shard definitions there), and create
> state-
> > > > > > > *<shardname>*.json for each shard with the replicas of that
> shard.
> > > > > > > Contention on anything replica related will be restricted to
> > > > replicas of
> > > > > > > the same shard.
> > > > > > > There will be more watches on ZooKeeper, they will trigger less
> > > > often and
> > > > > > > less data will be read each time. Also less data to
> > > > compress/uncompress
> > > > > > > each time state.json is written or read (when so configured).
> > > > > > >
> > > > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > > > >
> > > > > > > SolrCloud started with a single clusterstate.json file
> describing
> > > all
> > > > > > > collections (removed in 9.0), then moved to per collection
> > > state.json
> > > > > > files
> > > > > > > for scalability reasons.
> > > > > > > Maybe the time has come to split that big blob further?
> > > > > > >
> > > > > > > Ilan
> > > > > > >
> > > > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > > > hossman_lucene@fucit.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > : I don't think this should prevent shipping a system that is
> > > > > > objectively
> > > > > > > > way
> > > > > > > > : simpler than the Overseer. Solr 10 will have both modes,
> no
> > > > matter
> > > > > > > what
> > > > > > > > : the default is. Changing the default makes it easier to
> remove
> > > > it in
> > > > > > > > Solr
> > > > > > > > : 11. The impact on ease of understanding SolrCloud in 11
> will
> > > be
> > > > > > > amazing!
> > > > > > > >
> > > > > > > > I'm not understanding yoru claim that changing a default from
> > > A(x)
> > > > to
> > > > > > > A(y)
> > > > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > > > >
> > > > > > > > You could change the default in 10.1, 10.2, etc... and it
> would
> > > > still
> > > > > > be
> > > > > > > > the same amount of effort to remove it in 11.0.
> > > > > > > >
> > > > > > > > No matter when you change the default, if the *option* to use
> > > A(x)
> > > > > > still
> > > > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > > > implementing
> > > > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0
> have
> > > > some
> > > > > > > > code/process/documentation enabling users to migrate their
> > > cluster
> > > > to
> > > > > > > > A(y)
> > > > > > > >
> > > > > > > >
> > > > > > > > -Hoss
> > > > > > > > http://www.lucidworks.com/
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> > > --
> > > http://www.needhamsoftware.com (work)
> > > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ilan Ginzburg <il...@gmail.com>.
The design doc:
https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit
Timothy Potter identifying the issue right away:
https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit?disco=AAAAHGxplOw
On Thu, Oct 2, 2025 at 12:09 AM Ishan Chattopadhyaya
<ic...@gmail.com> wrote:
>
> Is there a SIP or design document for this feature (no overseer mode),
> please?
> I'm familiar with the traditional overseer based mode, the PRS mode, not
> very familiar with the DSU thing that Ilan + colleagues introduced. But, I
> can't find enough design notes for the no overseer mode.
>
> On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
>
> > Sounds like an experimental feature that could end up changing (to fix this
> > issue with replica count). While we want feedback from real users, this
> > discussion makes me think it's just not ready for default status yet.
> >
> > On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
> > wrote:
> >
> > > > Let's imagine (quite realistically) that I'm updating our upgrade notes
> > > and
> > > > providing advice for users to choose the Overseer or not. Given the
> > > > benefits & risks, at what replica count threshold (for the biggest
> > > > collection) would you advise continued use of the Overseer?
> > >
> > > Even assuming we can figure out a value on 'N' that sounds reasonable,
> > > can be shown to be stable in load testing, etc....is that enough to
> > > recommend "non-overseer" mode?
> > >
> > > Sizing (which includes replica-count) is notoriously a guess-and-check
> > > process in Solr. And even for users who have done everything right
> > > and dialed in their replica-count with some benchmarking - what
> > > happens when their requirements change and they need to add replicas
> > > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > > users to switch back to the overseer, or do they have to risk
> > > instability going forward?
> > >
> > > I guess I'm worried about basing recommendations on a factor like
> > > replica-count which has a tendency to drift over time, if the decision
> > > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > > Not 100% sure that's the case here, but that's my suspicion based on a
> > > hazy recollection of some past discussions.
> > >
> > > Best,
> > >
> > > Jason
> > >
> > > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> > wrote:
> > > >
> > > > It's hard to provide a recommended threshold on collection size for
> > > > distributed mode.
> > > > I didn't run tests and it obviously depends on the number of nodes in
> > the
> > > > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > > > that below a couple hundred replicas total for a collection it should
> > be
> > > ok.
> > > > When a Solr node starts it marks all its replicas DOWN before marking
> > > them
> > > > ACTIVE. If PRS is not used this could take a long time with distributed
> > > > mode and be slower than Overseer due to the lack of batching of
> > updates.
> > > >
> > > > Indexing and query performance is obviously not impacted by distributed
> > > > mode or Overseer performance, unless shard split performance is
> > > considered
> > > > part of indexing performance.
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > > wrote:
> > > >
> > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > > notes and
> > > > > providing advice for users to choose the Overseer or not. Given the
> > > > > benefits & risks, at what replica count threshold (for the biggest
> > > > > collection) would you advise continued use of the Overseer?
> > > > >
> > > > > Sufficient stability is the characteristic I'm looking for in the
> > above
> > > > > question, not performance. I understand that it'd be ideal if the
> > > > > cluster's state was structured differently to improve performance of
> > > > > certain administrative operations, but performance is not a
> > > requirement for
> > > > > stability. The most important performance considerations our users
> > > have
> > > > > relate to index & query. There's a basic assumption that nodes can
> > > restart
> > > > > in a "reasonable time"... maybe you'd care to try to define that. I
> > > think
> > > > > your recommendations around restructuring the cluster state would
> > > > > ultimately impact the performance of restarts and some other
> > > administrative
> > > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > > >
> > > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Distributed mode doesn't behave nicely when there are many
> > concurrent
> > > > > > updates to a given collection's state.json.
> > > > > >
> > > > > > I'd recommend *against* making it the default at this time.
> > > > > >
> > > > > > The "root cause" is the presence of replica specific information in
> > > > > > state.json. In addition to relatively rare cases of changes to the
> > > > > sharding
> > > > > > of the collection, state.json is updated when replicas are created
> > or
> > > > > > destroyed or moved or have their properties changed, and when PRS
> > is
> > > not
> > > > > > used, also when replicas change state (which happens a lot when a
> > > Solr
> > > > > node
> > > > > > restarts for example).
> > > > > >
> > > > > > Therefore before making distributed mode the default, something has
> > > to be
> > > > > > done.
> > > > > > As Pierre suggests, redesign Collection API operations that require
> > > > > > multiple updates to be more efficient and group them when executing
> > > in
> > > > > > distributed mode. Also make sure that smaller operations that
> > happen
> > > > > > concurrently are efficient enough.
> > > > > > Another option is to remove replica information from state.json
> > (keep
> > > > > > collection metadata and shard definitions there), and create state-
> > > > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > > > Contention on anything replica related will be restricted to
> > > replicas of
> > > > > > the same shard.
> > > > > > There will be more watches on ZooKeeper, they will trigger less
> > > often and
> > > > > > less data will be read each time. Also less data to
> > > compress/uncompress
> > > > > > each time state.json is written or read (when so configured).
> > > > > >
> > > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > > >
> > > > > > SolrCloud started with a single clusterstate.json file describing
> > all
> > > > > > collections (removed in 9.0), then moved to per collection
> > state.json
> > > > > files
> > > > > > for scalability reasons.
> > > > > > Maybe the time has come to split that big blob further?
> > > > > >
> > > > > > Ilan
> > > > > >
> > > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > > hossman_lucene@fucit.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > : I don't think this should prevent shipping a system that is
> > > > > objectively
> > > > > > > way
> > > > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> > > matter
> > > > > > what
> > > > > > > : the default is. Changing the default makes it easier to remove
> > > it in
> > > > > > > Solr
> > > > > > > : 11. The impact on ease of understanding SolrCloud in 11 will
> > be
> > > > > > amazing!
> > > > > > >
> > > > > > > I'm not understanding yoru claim that changing a default from
> > A(x)
> > > to
> > > > > > A(y)
> > > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > > >
> > > > > > > You could change the default in 10.1, 10.2, etc... and it would
> > > still
> > > > > be
> > > > > > > the same amount of effort to remove it in 11.0.
> > > > > > >
> > > > > > > No matter when you change the default, if the *option* to use
> > A(x)
> > > > > still
> > > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > > implementing
> > > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> > > some
> > > > > > > code/process/documentation enabling users to migrate their
> > cluster
> > > to
> > > > > > > A(y)
> > > > > > >
> > > > > > >
> > > > > > > -Hoss
> > > > > > > http://www.lucidworks.com/
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Is there a SIP or design document for this feature (no overseer mode),
please?
I'm familiar with the traditional overseer based mode, the PRS mode, not
very familiar with the DSU thing that Ilan + colleagues introduced. But, I
can't find enough design notes for the no overseer mode.
On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
> Sounds like an experimental feature that could end up changing (to fix this
> issue with replica count). While we want feedback from real users, this
> discussion makes me think it's just not ready for default status yet.
>
> On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
> > > Let's imagine (quite realistically) that I'm updating our upgrade notes
> > and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> >
> > Even assuming we can figure out a value on 'N' that sounds reasonable,
> > can be shown to be stable in load testing, etc....is that enough to
> > recommend "non-overseer" mode?
> >
> > Sizing (which includes replica-count) is notoriously a guess-and-check
> > process in Solr. And even for users who have done everything right
> > and dialed in their replica-count with some benchmarking - what
> > happens when their requirements change and they need to add replicas
> > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > users to switch back to the overseer, or do they have to risk
> > instability going forward?
> >
> > I guess I'm worried about basing recommendations on a factor like
> > replica-count which has a tendency to drift over time, if the decision
> > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > Not 100% sure that's the case here, but that's my suspicion based on a
> > hazy recollection of some past discussions.
> >
> > Best,
> >
> > Jason
> >
> > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > It's hard to provide a recommended threshold on collection size for
> > > distributed mode.
> > > I didn't run tests and it obviously depends on the number of nodes in
> the
> > > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > > that below a couple hundred replicas total for a collection it should
> be
> > ok.
> > > When a Solr node starts it marks all its replicas DOWN before marking
> > them
> > > ACTIVE. If PRS is not used this could take a long time with distributed
> > > mode and be slower than Overseer due to the lack of batching of
> updates.
> > >
> > > Indexing and query performance is obviously not impacted by distributed
> > > mode or Overseer performance, unless shard split performance is
> > considered
> > > part of indexing performance.
> > >
> > > Ilan
> > >
> > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > wrote:
> > >
> > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > notes and
> > > > providing advice for users to choose the Overseer or not. Given the
> > > > benefits & risks, at what replica count threshold (for the biggest
> > > > collection) would you advise continued use of the Overseer?
> > > >
> > > > Sufficient stability is the characteristic I'm looking for in the
> above
> > > > question, not performance. I understand that it'd be ideal if the
> > > > cluster's state was structured differently to improve performance of
> > > > certain administrative operations, but performance is not a
> > requirement for
> > > > stability. The most important performance considerations our users
> > have
> > > > relate to index & query. There's a basic assumption that nodes can
> > restart
> > > > in a "reasonable time"... maybe you'd care to try to define that. I
> > think
> > > > your recommendations around restructuring the cluster state would
> > > > ultimately impact the performance of restarts and some other
> > administrative
> > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > >
> > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> > wrote:
> > > >
> > > > > Distributed mode doesn't behave nicely when there are many
> concurrent
> > > > > updates to a given collection's state.json.
> > > > >
> > > > > I'd recommend *against* making it the default at this time.
> > > > >
> > > > > The "root cause" is the presence of replica specific information in
> > > > > state.json. In addition to relatively rare cases of changes to the
> > > > sharding
> > > > > of the collection, state.json is updated when replicas are created
> or
> > > > > destroyed or moved or have their properties changed, and when PRS
> is
> > not
> > > > > used, also when replicas change state (which happens a lot when a
> > Solr
> > > > node
> > > > > restarts for example).
> > > > >
> > > > > Therefore before making distributed mode the default, something has
> > to be
> > > > > done.
> > > > > As Pierre suggests, redesign Collection API operations that require
> > > > > multiple updates to be more efficient and group them when executing
> > in
> > > > > distributed mode. Also make sure that smaller operations that
> happen
> > > > > concurrently are efficient enough.
> > > > > Another option is to remove replica information from state.json
> (keep
> > > > > collection metadata and shard definitions there), and create state-
> > > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > > Contention on anything replica related will be restricted to
> > replicas of
> > > > > the same shard.
> > > > > There will be more watches on ZooKeeper, they will trigger less
> > often and
> > > > > less data will be read each time. Also less data to
> > compress/uncompress
> > > > > each time state.json is written or read (when so configured).
> > > > >
> > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > >
> > > > > SolrCloud started with a single clusterstate.json file describing
> all
> > > > > collections (removed in 9.0), then moved to per collection
> state.json
> > > > files
> > > > > for scalability reasons.
> > > > > Maybe the time has come to split that big blob further?
> > > > >
> > > > > Ilan
> > > > >
> > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > hossman_lucene@fucit.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > : I don't think this should prevent shipping a system that is
> > > > objectively
> > > > > > way
> > > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> > matter
> > > > > what
> > > > > > : the default is. Changing the default makes it easier to remove
> > it in
> > > > > > Solr
> > > > > > : 11. The impact on ease of understanding SolrCloud in 11 will
> be
> > > > > amazing!
> > > > > >
> > > > > > I'm not understanding yoru claim that changing a default from
> A(x)
> > to
> > > > > A(y)
> > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > >
> > > > > > You could change the default in 10.1, 10.2, etc... and it would
> > still
> > > > be
> > > > > > the same amount of effort to remove it in 11.0.
> > > > > >
> > > > > > No matter when you change the default, if the *option* to use
> A(x)
> > > > still
> > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > implementing
> > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> > some
> > > > > > code/process/documentation enabling users to migrate their
> cluster
> > to
> > > > > > A(y)
> > > > > >
> > > > > >
> > > > > > -Hoss
> > > > > > http://www.lucidworks.com/
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Gus Heck <gu...@gmail.com>.
Sounds like an experimental feature that could end up changing (to fix this
issue with replica count). While we want feedback from real users, this
discussion makes me think it's just not ready for default status yet.
On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> > Let's imagine (quite realistically) that I'm updating our upgrade notes
> and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
>
> Even assuming we can figure out a value on 'N' that sounds reasonable,
> can be shown to be stable in load testing, etc....is that enough to
> recommend "non-overseer" mode?
>
> Sizing (which includes replica-count) is notoriously a guess-and-check
> process in Solr. And even for users who have done everything right
> and dialed in their replica-count with some benchmarking - what
> happens when their requirements change and they need to add replicas
> to (e.g.) accommodate a higher QPS. Is there an easy way for those
> users to switch back to the overseer, or do they have to risk
> instability going forward?
>
> I guess I'm worried about basing recommendations on a factor like
> replica-count which has a tendency to drift over time, if the decision
> itself (i.e. overseer or not) is difficult to reverse after the fact.
> Not 100% sure that's the case here, but that's my suspicion based on a
> hazy recollection of some past discussions.
>
> Best,
>
> Jason
>
> On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > It's hard to provide a recommended threshold on collection size for
> > distributed mode.
> > I didn't run tests and it obviously depends on the number of nodes in the
> > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > that below a couple hundred replicas total for a collection it should be
> ok.
> > When a Solr node starts it marks all its replicas DOWN before marking
> them
> > ACTIVE. If PRS is not used this could take a long time with distributed
> > mode and be slower than Overseer due to the lack of batching of updates.
> >
> > Indexing and query performance is obviously not impacted by distributed
> > mode or Overseer performance, unless shard split performance is
> considered
> > part of indexing performance.
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> > >
> > > Sufficient stability is the characteristic I'm looking for in the above
> > > question, not performance. I understand that it'd be ideal if the
> > > cluster's state was structured differently to improve performance of
> > > certain administrative operations, but performance is not a
> requirement for
> > > stability. The most important performance considerations our users
> have
> > > relate to index & query. There's a basic assumption that nodes can
> restart
> > > in a "reasonable time"... maybe you'd care to try to define that. I
> think
> > > your recommendations around restructuring the cluster state would
> > > ultimately impact the performance of restarts and some other
> administrative
> > > scenarios but shouldn't be a prerequisite for a stable system.
> > >
> > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > > Distributed mode doesn't behave nicely when there are many concurrent
> > > > updates to a given collection's state.json.
> > > >
> > > > I'd recommend *against* making it the default at this time.
> > > >
> > > > The "root cause" is the presence of replica specific information in
> > > > state.json. In addition to relatively rare cases of changes to the
> > > sharding
> > > > of the collection, state.json is updated when replicas are created or
> > > > destroyed or moved or have their properties changed, and when PRS is
> not
> > > > used, also when replicas change state (which happens a lot when a
> Solr
> > > node
> > > > restarts for example).
> > > >
> > > > Therefore before making distributed mode the default, something has
> to be
> > > > done.
> > > > As Pierre suggests, redesign Collection API operations that require
> > > > multiple updates to be more efficient and group them when executing
> in
> > > > distributed mode. Also make sure that smaller operations that happen
> > > > concurrently are efficient enough.
> > > > Another option is to remove replica information from state.json (keep
> > > > collection metadata and shard definitions there), and create state-
> > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > Contention on anything replica related will be restricted to
> replicas of
> > > > the same shard.
> > > > There will be more watches on ZooKeeper, they will trigger less
> often and
> > > > less data will be read each time. Also less data to
> compress/uncompress
> > > > each time state.json is written or read (when so configured).
> > > >
> > > > Throttling goes against making SolrCloud as fast as we can.
> > > >
> > > > SolrCloud started with a single clusterstate.json file describing all
> > > > collections (removed in 9.0), then moved to per collection state.json
> > > files
> > > > for scalability reasons.
> > > > Maybe the time has come to split that big blob further?
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > hossman_lucene@fucit.org
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > : I don't think this should prevent shipping a system that is
> > > objectively
> > > > > way
> > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> matter
> > > > what
> > > > > : the default is. Changing the default makes it easier to remove
> it in
> > > > > Solr
> > > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > > amazing!
> > > > >
> > > > > I'm not understanding yoru claim that changing a default from A(x)
> to
> > > > A(y)
> > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > >
> > > > > You could change the default in 10.1, 10.2, etc... and it would
> still
> > > be
> > > > > the same amount of effort to remove it in 11.0.
> > > > >
> > > > > No matter when you change the default, if the *option* to use A(x)
> > > still
> > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > implementing
> > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> some
> > > > > code/process/documentation enabling users to migrate their cluster
> to
> > > > > A(y)
> > > > >
> > > > >
> > > > > -Hoss
> > > > > http://www.lucidworks.com/
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Jason Gerlowski <ge...@gmail.com>.
> Let's imagine (quite realistically) that I'm updating our upgrade notes and
> providing advice for users to choose the Overseer or not. Given the
> benefits & risks, at what replica count threshold (for the biggest
> collection) would you advise continued use of the Overseer?
Even assuming we can figure out a value on 'N' that sounds reasonable,
can be shown to be stable in load testing, etc....is that enough to
recommend "non-overseer" mode?
Sizing (which includes replica-count) is notoriously a guess-and-check
process in Solr. And even for users who have done everything right
and dialed in their replica-count with some benchmarking - what
happens when their requirements change and they need to add replicas
to (e.g.) accommodate a higher QPS. Is there an easy way for those
users to switch back to the overseer, or do they have to risk
instability going forward?
I guess I'm worried about basing recommendations on a factor like
replica-count which has a tendency to drift over time, if the decision
itself (i.e. overseer or not) is difficult to reverse after the fact.
Not 100% sure that's the case here, but that's my suspicion based on a
hazy recollection of some past discussions.
Best,
Jason
On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> It's hard to provide a recommended threshold on collection size for
> distributed mode.
> I didn't run tests and it obviously depends on the number of nodes in the
> cluster and how fast everything (including ZooKeeper) runs, but I'd say
> that below a couple hundred replicas total for a collection it should be ok.
> When a Solr node starts it marks all its replicas DOWN before marking them
> ACTIVE. If PRS is not used this could take a long time with distributed
> mode and be slower than Overseer due to the lack of batching of updates.
>
> Indexing and query performance is obviously not impacted by distributed
> mode or Overseer performance, unless shard split performance is considered
> part of indexing performance.
>
> Ilan
>
> On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org> wrote:
>
> > Let's imagine (quite realistically) that I'm updating our upgrade notes and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
> >
> > Sufficient stability is the characteristic I'm looking for in the above
> > question, not performance. I understand that it'd be ideal if the
> > cluster's state was structured differently to improve performance of
> > certain administrative operations, but performance is not a requirement for
> > stability. The most important performance considerations our users have
> > relate to index & query. There's a basic assumption that nodes can restart
> > in a "reasonable time"... maybe you'd care to try to define that. I think
> > your recommendations around restructuring the cluster state would
> > ultimately impact the performance of restarts and some other administrative
> > scenarios but shouldn't be a prerequisite for a stable system.
> >
> > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > > Distributed mode doesn't behave nicely when there are many concurrent
> > > updates to a given collection's state.json.
> > >
> > > I'd recommend *against* making it the default at this time.
> > >
> > > The "root cause" is the presence of replica specific information in
> > > state.json. In addition to relatively rare cases of changes to the
> > sharding
> > > of the collection, state.json is updated when replicas are created or
> > > destroyed or moved or have their properties changed, and when PRS is not
> > > used, also when replicas change state (which happens a lot when a Solr
> > node
> > > restarts for example).
> > >
> > > Therefore before making distributed mode the default, something has to be
> > > done.
> > > As Pierre suggests, redesign Collection API operations that require
> > > multiple updates to be more efficient and group them when executing in
> > > distributed mode. Also make sure that smaller operations that happen
> > > concurrently are efficient enough.
> > > Another option is to remove replica information from state.json (keep
> > > collection metadata and shard definitions there), and create state-
> > > *<shardname>*.json for each shard with the replicas of that shard.
> > > Contention on anything replica related will be restricted to replicas of
> > > the same shard.
> > > There will be more watches on ZooKeeper, they will trigger less often and
> > > less data will be read each time. Also less data to compress/uncompress
> > > each time state.json is written or read (when so configured).
> > >
> > > Throttling goes against making SolrCloud as fast as we can.
> > >
> > > SolrCloud started with a single clusterstate.json file describing all
> > > collections (removed in 9.0), then moved to per collection state.json
> > files
> > > for scalability reasons.
> > > Maybe the time has come to split that big blob further?
> > >
> > > Ilan
> > >
> > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > hossman_lucene@fucit.org
> > > >
> > > wrote:
> > >
> > > >
> > > > : I don't think this should prevent shipping a system that is
> > objectively
> > > > way
> > > > : simpler than the Overseer. Solr 10 will have both modes, no matter
> > > what
> > > > : the default is. Changing the default makes it easier to remove it in
> > > > Solr
> > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > amazing!
> > > >
> > > > I'm not understanding yoru claim that changing a default from A(x) to
> > > A(y)
> > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > >
> > > > You could change the default in 10.1, 10.2, etc... and it would still
> > be
> > > > the same amount of effort to remove it in 11.0.
> > > >
> > > > No matter when you change the default, if the *option* to use A(x)
> > still
> > > > exists in all versions < 11.0, then any "removal" of the code
> > > implementing
> > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> > > > code/process/documentation enabling users to migrate their cluster to
> > > > A(y)
> > > >
> > > >
> > > > -Hoss
> > > > http://www.lucidworks.com/
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ilan Ginzburg <il...@gmail.com>.
It's hard to provide a recommended threshold on collection size for
distributed mode.
I didn't run tests and it obviously depends on the number of nodes in the
cluster and how fast everything (including ZooKeeper) runs, but I'd say
that below a couple hundred replicas total for a collection it should be ok.
When a Solr node starts it marks all its replicas DOWN before marking them
ACTIVE. If PRS is not used this could take a long time with distributed
mode and be slower than Overseer due to the lack of batching of updates.
Indexing and query performance is obviously not impacted by distributed
mode or Overseer performance, unless shard split performance is considered
part of indexing performance.
Ilan
On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org> wrote:
> Let's imagine (quite realistically) that I'm updating our upgrade notes and
> providing advice for users to choose the Overseer or not. Given the
> benefits & risks, at what replica count threshold (for the biggest
> collection) would you advise continued use of the Overseer?
>
> Sufficient stability is the characteristic I'm looking for in the above
> question, not performance. I understand that it'd be ideal if the
> cluster's state was structured differently to improve performance of
> certain administrative operations, but performance is not a requirement for
> stability. The most important performance considerations our users have
> relate to index & query. There's a basic assumption that nodes can restart
> in a "reasonable time"... maybe you'd care to try to define that. I think
> your recommendations around restructuring the cluster state would
> ultimately impact the performance of restarts and some other administrative
> scenarios but shouldn't be a prerequisite for a stable system.
>
> On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> > Distributed mode doesn't behave nicely when there are many concurrent
> > updates to a given collection's state.json.
> >
> > I'd recommend *against* making it the default at this time.
> >
> > The "root cause" is the presence of replica specific information in
> > state.json. In addition to relatively rare cases of changes to the
> sharding
> > of the collection, state.json is updated when replicas are created or
> > destroyed or moved or have their properties changed, and when PRS is not
> > used, also when replicas change state (which happens a lot when a Solr
> node
> > restarts for example).
> >
> > Therefore before making distributed mode the default, something has to be
> > done.
> > As Pierre suggests, redesign Collection API operations that require
> > multiple updates to be more efficient and group them when executing in
> > distributed mode. Also make sure that smaller operations that happen
> > concurrently are efficient enough.
> > Another option is to remove replica information from state.json (keep
> > collection metadata and shard definitions there), and create state-
> > *<shardname>*.json for each shard with the replicas of that shard.
> > Contention on anything replica related will be restricted to replicas of
> > the same shard.
> > There will be more watches on ZooKeeper, they will trigger less often and
> > less data will be read each time. Also less data to compress/uncompress
> > each time state.json is written or read (when so configured).
> >
> > Throttling goes against making SolrCloud as fast as we can.
> >
> > SolrCloud started with a single clusterstate.json file describing all
> > collections (removed in 9.0), then moved to per collection state.json
> files
> > for scalability reasons.
> > Maybe the time has come to split that big blob further?
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> hossman_lucene@fucit.org
> > >
> > wrote:
> >
> > >
> > > : I don't think this should prevent shipping a system that is
> objectively
> > > way
> > > : simpler than the Overseer. Solr 10 will have both modes, no matter
> > what
> > > : the default is. Changing the default makes it easier to remove it in
> > > Solr
> > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > amazing!
> > >
> > > I'm not understanding yoru claim that changing a default from A(x) to
> > A(y)
> > > in 10.0 makes removing A(x) in 11.0 easier?
> > >
> > > You could change the default in 10.1, 10.2, etc... and it would still
> be
> > > the same amount of effort to remove it in 11.0.
> > >
> > > No matter when you change the default, if the *option* to use A(x)
> still
> > > exists in all versions < 11.0, then any "removal" of the code
> > implementing
> > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> > > code/process/documentation enabling users to migrate their cluster to
> > > A(y)
> > >
> > >
> > > -Hoss
> > > http://www.lucidworks.com/
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by David Smiley <ds...@apache.org>.
Let's imagine (quite realistically) that I'm updating our upgrade notes and
providing advice for users to choose the Overseer or not. Given the
benefits & risks, at what replica count threshold (for the biggest
collection) would you advise continued use of the Overseer?
Sufficient stability is the characteristic I'm looking for in the above
question, not performance. I understand that it'd be ideal if the
cluster's state was structured differently to improve performance of
certain administrative operations, but performance is not a requirement for
stability. The most important performance considerations our users have
relate to index & query. There's a basic assumption that nodes can restart
in a "reasonable time"... maybe you'd care to try to define that. I think
your recommendations around restructuring the cluster state would
ultimately impact the performance of restarts and some other administrative
scenarios but shouldn't be a prerequisite for a stable system.
On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com> wrote:
> Distributed mode doesn't behave nicely when there are many concurrent
> updates to a given collection's state.json.
>
> I'd recommend *against* making it the default at this time.
>
> The "root cause" is the presence of replica specific information in
> state.json. In addition to relatively rare cases of changes to the sharding
> of the collection, state.json is updated when replicas are created or
> destroyed or moved or have their properties changed, and when PRS is not
> used, also when replicas change state (which happens a lot when a Solr node
> restarts for example).
>
> Therefore before making distributed mode the default, something has to be
> done.
> As Pierre suggests, redesign Collection API operations that require
> multiple updates to be more efficient and group them when executing in
> distributed mode. Also make sure that smaller operations that happen
> concurrently are efficient enough.
> Another option is to remove replica information from state.json (keep
> collection metadata and shard definitions there), and create state-
> *<shardname>*.json for each shard with the replicas of that shard.
> Contention on anything replica related will be restricted to replicas of
> the same shard.
> There will be more watches on ZooKeeper, they will trigger less often and
> less data will be read each time. Also less data to compress/uncompress
> each time state.json is written or read (when so configured).
>
> Throttling goes against making SolrCloud as fast as we can.
>
> SolrCloud started with a single clusterstate.json file describing all
> collections (removed in 9.0), then moved to per collection state.json files
> for scalability reasons.
> Maybe the time has come to split that big blob further?
>
> Ilan
>
> On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <hossman_lucene@fucit.org
> >
> wrote:
>
> >
> > : I don't think this should prevent shipping a system that is objectively
> > way
> > : simpler than the Overseer. Solr 10 will have both modes, no matter
> what
> > : the default is. Changing the default makes it easier to remove it in
> > Solr
> > : 11. The impact on ease of understanding SolrCloud in 11 will be
> amazing!
> >
> > I'm not understanding yoru claim that changing a default from A(x) to
> A(y)
> > in 10.0 makes removing A(x) in 11.0 easier?
> >
> > You could change the default in 10.1, 10.2, etc... and it would still be
> > the same amount of effort to remove it in 11.0.
> >
> > No matter when you change the default, if the *option* to use A(x) still
> > exists in all versions < 11.0, then any "removal" of the code
> implementing
> > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> > code/process/documentation enabling users to migrate their cluster to
> > A(y)
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ilan Ginzburg <il...@gmail.com>.
Distributed mode doesn't behave nicely when there are many concurrent
updates to a given collection's state.json.
I'd recommend *against* making it the default at this time.
The "root cause" is the presence of replica specific information in
state.json. In addition to relatively rare cases of changes to the sharding
of the collection, state.json is updated when replicas are created or
destroyed or moved or have their properties changed, and when PRS is not
used, also when replicas change state (which happens a lot when a Solr node
restarts for example).
Therefore before making distributed mode the default, something has to be
done.
As Pierre suggests, redesign Collection API operations that require
multiple updates to be more efficient and group them when executing in
distributed mode. Also make sure that smaller operations that happen
concurrently are efficient enough.
Another option is to remove replica information from state.json (keep
collection metadata and shard definitions there), and create state-
*<shardname>*.json for each shard with the replicas of that shard.
Contention on anything replica related will be restricted to replicas of
the same shard.
There will be more watches on ZooKeeper, they will trigger less often and
less data will be read each time. Also less data to compress/uncompress
each time state.json is written or read (when so configured).
Throttling goes against making SolrCloud as fast as we can.
SolrCloud started with a single clusterstate.json file describing all
collections (removed in 9.0), then moved to per collection state.json files
for scalability reasons.
Maybe the time has come to split that big blob further?
Ilan
On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : I don't think this should prevent shipping a system that is objectively
> way
> : simpler than the Overseer. Solr 10 will have both modes, no matter what
> : the default is. Changing the default makes it easier to remove it in
> Solr
> : 11. The impact on ease of understanding SolrCloud in 11 will be amazing!
>
> I'm not understanding yoru claim that changing a default from A(x) to A(y)
> in 10.0 makes removing A(x) in 11.0 easier?
>
> You could change the default in 10.1, 10.2, etc... and it would still be
> the same amount of effort to remove it in 11.0.
>
> No matter when you change the default, if the *option* to use A(x) still
> exists in all versions < 11.0, then any "removal" of the code implementing
> A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> code/process/documentation enabling users to migrate their cluster to
> A(y)
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Chris Hostetter <ho...@fucit.org>.
: I don't think this should prevent shipping a system that is objectively way
: simpler than the Overseer. Solr 10 will have both modes, no matter what
: the default is. Changing the default makes it easier to remove it in Solr
: 11. The impact on ease of understanding SolrCloud in 11 will be amazing!
I'm not understanding yoru claim that changing a default from A(x) to A(y)
in 10.0 makes removing A(x) in 11.0 easier?
You could change the default in 10.1, 10.2, etc... and it would still be
the same amount of effort to remove it in 11.0.
No matter when you change the default, if the *option* to use A(x) still
exists in all versions < 11.0, then any "removal" of the code implementing
A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
code/process/documentation enabling users to migrate their cluster to
A(y)
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Pierre Salagnac <pi...@gmail.com>.
Not sure we should workaround an imperfect design by adding throttling.
Plus this throttling would be at the cluster level, while the issue with
distributed mode is when concurrent updates are done for the same
collection.
I agree not having a single node that does all the updates looks much
better on paper. But unfortunately, it seems to be not so easy in practice.
AFAIK distributed mode hasn't been used by anyone at scale yet. There are
many changes required before we can consider it as stable.
For example, discussing this specific issue when deleting a big collection,
there is no reason each node removes its replicas from the ZK collection
state. We want the collection to be removed, so why not just remove it from
Zookeeper with a single operation? I think some admin operations must be
redesigned. We can't assume what scales well with the overseer will scale
well with "no-overseer" mode.
Unfortunately, I don't have enough experience with running distributed
updates to say whether other major problems exist.
Making it the default in Solr 10 would mean we are confident it works well.
Personally, I'm not confident yet.
Best,
Pierre
Le lun. 29 sept. 2025 à 15:36, David Smiley <ds...@apache.org> a écrit :
> About the particular scenario: couldn't we just limit the number of
> in-flight requests? Cluster admin commands use HttpShardHandler, and it
> has an executor defaulting to Integer.MAX_VALUE. This is configurable in
> solr.xml with "maximumPoolSize" but I'm not certain if that same instance
> of ShardHandler is also used for cluster admin commands. We ought to have
> separately configurable ones.
>
> I don't think this should prevent shipping a system that is objectively way
> simpler than the Overseer. Solr 10 will have both modes, no matter what
> the default is. Changing the default makes it easier to remove it in Solr
> 11. The impact on ease of understanding SolrCloud in 11 will be amazing!
> It can't be seen now because we can't remove the needless ZNodeProps admin
> command conversions that exist simply because we put commands into ZK
> queues.
>
> ~ David
>
> On Mon, Sep 29, 2025 at 9:00 AM Pierre Salagnac <ps...@apache.org>
> wrote:
>
> > Indeed, we're having some issues with "no-overseer" mode and we decided
> to
> > hold off deployment of this feature for now.
> >
> > This is mostly because of a design flaw in distributed cluster updates.
> > When a collection reaches a given size, we end up having many "writers"
> > that want to update the state.json of this collection concurrently. Since
> > they all do Zookeeper "check-and-write" operations using the version
> > number, there is no data corruption, but this puts a huge load on the
> > cluster.
> >
> > This issue has a major impact when deleting a large collection (1000+
> > cores). The collection API sends 1000+ UNLOAD core admin requests that
> are
> > processed concurrently, and each one wants to update the 'state.json'
> file
> > of the collection. We enter a tidal wave of writes operations that are
> > rejected by Zookeeper.
> >
> > This mode probably works pretty well and scales better when there are
> many
> > small collections in the cluster. Updates to these collections can then
> be
> > done concurrently since the probability of having write conflicts is low.
> > That's the opposite for big collections, where it seems the overseer
> scales
> > better.
> >
> > Best,
> > Pierre
> >
> >
> > Le ven. 26 sept. 2025 à 15:37, Jason Gerlowski <ge...@gmail.com> a
> > écrit :
> >
> > > In the latest Virtual Community Meetup, Pierre and Bruno shared that
> > > they recently enabled "no-overseer" mode (really need a better name!)
> > > at their workplace in production and were hitting some stability
> > > issues. Their anecdotes give me at least some "pause" about whether
> > > this is really ready to be the default, or whether it needs a bit more
> > > time to bake.
> > >
> > > Any chance either of them are watching this thread and can provide an
> > > update or their own 2c here?
> > >
> > > Best,
> > >
> > > Jason
> > >
> > > On Wed, Aug 20, 2025 at 6:35 PM David Smiley <ds...@apache.org>
> wrote:
> > > >
> > > > Update: I proposed the Solr version go into cluster properties[1] on
> > ZK
> > > > initialization but Houston pushed back on that approach. The other
> > > > approach is to rely on the "least Solr version" as returned by the
> Solr
> > > > version being added to live nodes[2]. However that's very dynamic
> and
> > > I'm
> > > > concerned about an old Solr version somehow joining an existing
> > cluster.
> > > > Rather than simply tell users "don't do that" (which we should do
> > > anyway),
> > > > I'm inclined to have Solr fail to join an existing cluster if doing
> so
> > > > would *lower* the least Solr version. Perhaps ignoring the final
> patch
> > > > version. I plan on updating that PR accordingly. The PR would have
> to
> > > go
> > > > into 9.x too, and thus such logic can't be enforced for older Solr
> > > > versions. Regardless, an upgrading user can choose the setting.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/SOLR-17664
> > > > [2] https://issues.apache.org/jira/browse/SOLR-17620
> > > >
> > > > On Tue, Jul 22, 2025 at 1:12 AM David Smiley <ds...@apache.org>
> > wrote:
> > > >
> > > > > On the epic of seeking the _eventual_ demise of the Overseer, I'm
> > > seeking
> > > > > to make it disabled for *new* SolrCloud clusters in Solr 10. --
> > > > > https://issues.apache.org/jira/browse/SOLR-17293
> > > > > The epic: https://issues.apache.org/jira/browse/SOLR-14927 (oddly
> no
> > > SIP
> > > > > but it has a doc anyway). I think it's sufficiently ready for the
> > > great
> > > > > majority of SolrCloud clusters. A cluster with a collection
> > > containing a
> > > > > thousand+ replicas might pose a performance concern on
> start/restart
> > > events
> > > > > due to independent replica state updates. Of course, with such a
> > > change,
> > > > > there will be a section in the upgrade page in the ref guide to
> > advise
> > > > > users who may opt to make an explicit choice.
> > > > >
> > > > > I don't love that the new mode doesn't have an elegant/clear way to
> > > refer
> > > > > to it. The best I've come up with is to say what it *isn't* -- it
> > > *isn't*
> > > > > the Overseer. "The Overseer is disabled". Awkwardly there are two
> > > > > undocumented solr.xml booleans, both a mouthful:
> > > > > distributedClusterStateUpdates and
> > > > > distributedCollectionConfigSetExecution. I propose instead that a
> > > single
> > > > > boolean cluster property be defined named "overseer" or
> > > "overseerEnabled".
> > > > > FYI the existing known cluster properties are defined here:
> > > > > org.apache.solr.common.cloud.ZkStateReader#KNOWN_CLUSTER_PROPS
> > > > > Even if such a boolean is agreeable... it raises the question of
> what
> > > > > should become of the "overseer" node role.
> > > > >
> > > > > ~ David Smiley
> > > > > Apache Lucene/Solr Search Developer
> > > > > http://www.linkedin.com/in/davidwsmiley
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by David Smiley <ds...@apache.org>.
About the particular scenario: couldn't we just limit the number of
in-flight requests? Cluster admin commands use HttpShardHandler, and it
has an executor defaulting to Integer.MAX_VALUE. This is configurable in
solr.xml with "maximumPoolSize" but I'm not certain if that same instance
of ShardHandler is also used for cluster admin commands. We ought to have
separately configurable ones.
I don't think this should prevent shipping a system that is objectively way
simpler than the Overseer. Solr 10 will have both modes, no matter what
the default is. Changing the default makes it easier to remove it in Solr
11. The impact on ease of understanding SolrCloud in 11 will be amazing!
It can't be seen now because we can't remove the needless ZNodeProps admin
command conversions that exist simply because we put commands into ZK
queues.
~ David
On Mon, Sep 29, 2025 at 9:00 AM Pierre Salagnac <ps...@apache.org>
wrote:
> Indeed, we're having some issues with "no-overseer" mode and we decided to
> hold off deployment of this feature for now.
>
> This is mostly because of a design flaw in distributed cluster updates.
> When a collection reaches a given size, we end up having many "writers"
> that want to update the state.json of this collection concurrently. Since
> they all do Zookeeper "check-and-write" operations using the version
> number, there is no data corruption, but this puts a huge load on the
> cluster.
>
> This issue has a major impact when deleting a large collection (1000+
> cores). The collection API sends 1000+ UNLOAD core admin requests that are
> processed concurrently, and each one wants to update the 'state.json' file
> of the collection. We enter a tidal wave of writes operations that are
> rejected by Zookeeper.
>
> This mode probably works pretty well and scales better when there are many
> small collections in the cluster. Updates to these collections can then be
> done concurrently since the probability of having write conflicts is low.
> That's the opposite for big collections, where it seems the overseer scales
> better.
>
> Best,
> Pierre
>
>
> Le ven. 26 sept. 2025 à 15:37, Jason Gerlowski <ge...@gmail.com> a
> écrit :
>
> > In the latest Virtual Community Meetup, Pierre and Bruno shared that
> > they recently enabled "no-overseer" mode (really need a better name!)
> > at their workplace in production and were hitting some stability
> > issues. Their anecdotes give me at least some "pause" about whether
> > this is really ready to be the default, or whether it needs a bit more
> > time to bake.
> >
> > Any chance either of them are watching this thread and can provide an
> > update or their own 2c here?
> >
> > Best,
> >
> > Jason
> >
> > On Wed, Aug 20, 2025 at 6:35 PM David Smiley <ds...@apache.org> wrote:
> > >
> > > Update: I proposed the Solr version go into cluster properties[1] on
> ZK
> > > initialization but Houston pushed back on that approach. The other
> > > approach is to rely on the "least Solr version" as returned by the Solr
> > > version being added to live nodes[2]. However that's very dynamic and
> > I'm
> > > concerned about an old Solr version somehow joining an existing
> cluster.
> > > Rather than simply tell users "don't do that" (which we should do
> > anyway),
> > > I'm inclined to have Solr fail to join an existing cluster if doing so
> > > would *lower* the least Solr version. Perhaps ignoring the final patch
> > > version. I plan on updating that PR accordingly. The PR would have to
> > go
> > > into 9.x too, and thus such logic can't be enforced for older Solr
> > > versions. Regardless, an upgrading user can choose the setting.
> > >
> > > [1] https://issues.apache.org/jira/browse/SOLR-17664
> > > [2] https://issues.apache.org/jira/browse/SOLR-17620
> > >
> > > On Tue, Jul 22, 2025 at 1:12 AM David Smiley <ds...@apache.org>
> wrote:
> > >
> > > > On the epic of seeking the _eventual_ demise of the Overseer, I'm
> > seeking
> > > > to make it disabled for *new* SolrCloud clusters in Solr 10. --
> > > > https://issues.apache.org/jira/browse/SOLR-17293
> > > > The epic: https://issues.apache.org/jira/browse/SOLR-14927 (oddly no
> > SIP
> > > > but it has a doc anyway). I think it's sufficiently ready for the
> > great
> > > > majority of SolrCloud clusters. A cluster with a collection
> > containing a
> > > > thousand+ replicas might pose a performance concern on start/restart
> > events
> > > > due to independent replica state updates. Of course, with such a
> > change,
> > > > there will be a section in the upgrade page in the ref guide to
> advise
> > > > users who may opt to make an explicit choice.
> > > >
> > > > I don't love that the new mode doesn't have an elegant/clear way to
> > refer
> > > > to it. The best I've come up with is to say what it *isn't* -- it
> > *isn't*
> > > > the Overseer. "The Overseer is disabled". Awkwardly there are two
> > > > undocumented solr.xml booleans, both a mouthful:
> > > > distributedClusterStateUpdates and
> > > > distributedCollectionConfigSetExecution. I propose instead that a
> > single
> > > > boolean cluster property be defined named "overseer" or
> > "overseerEnabled".
> > > > FYI the existing known cluster properties are defined here:
> > > > org.apache.solr.common.cloud.ZkStateReader#KNOWN_CLUSTER_PROPS
> > > > Even if such a boolean is agreeable... it raises the question of what
> > > > should become of the "overseer" node role.
> > > >
> > > > ~ David Smiley
> > > > Apache Lucene/Solr Search Developer
> > > > http://www.linkedin.com/in/davidwsmiley
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Pierre Salagnac <ps...@apache.org>.
Indeed, we're having some issues with "no-overseer" mode and we decided to
hold off deployment of this feature for now.
This is mostly because of a design flaw in distributed cluster updates.
When a collection reaches a given size, we end up having many "writers"
that want to update the state.json of this collection concurrently. Since
they all do Zookeeper "check-and-write" operations using the version
number, there is no data corruption, but this puts a huge load on the
cluster.
This issue has a major impact when deleting a large collection (1000+
cores). The collection API sends 1000+ UNLOAD core admin requests that are
processed concurrently, and each one wants to update the 'state.json' file
of the collection. We enter a tidal wave of writes operations that are
rejected by Zookeeper.
This mode probably works pretty well and scales better when there are many
small collections in the cluster. Updates to these collections can then be
done concurrently since the probability of having write conflicts is low.
That's the opposite for big collections, where it seems the overseer scales
better.
Best,
Pierre
Le ven. 26 sept. 2025 à 15:37, Jason Gerlowski <ge...@gmail.com> a
écrit :
> In the latest Virtual Community Meetup, Pierre and Bruno shared that
> they recently enabled "no-overseer" mode (really need a better name!)
> at their workplace in production and were hitting some stability
> issues. Their anecdotes give me at least some "pause" about whether
> this is really ready to be the default, or whether it needs a bit more
> time to bake.
>
> Any chance either of them are watching this thread and can provide an
> update or their own 2c here?
>
> Best,
>
> Jason
>
> On Wed, Aug 20, 2025 at 6:35 PM David Smiley <ds...@apache.org> wrote:
> >
> > Update: I proposed the Solr version go into cluster properties[1] on ZK
> > initialization but Houston pushed back on that approach. The other
> > approach is to rely on the "least Solr version" as returned by the Solr
> > version being added to live nodes[2]. However that's very dynamic and
> I'm
> > concerned about an old Solr version somehow joining an existing cluster.
> > Rather than simply tell users "don't do that" (which we should do
> anyway),
> > I'm inclined to have Solr fail to join an existing cluster if doing so
> > would *lower* the least Solr version. Perhaps ignoring the final patch
> > version. I plan on updating that PR accordingly. The PR would have to
> go
> > into 9.x too, and thus such logic can't be enforced for older Solr
> > versions. Regardless, an upgrading user can choose the setting.
> >
> > [1] https://issues.apache.org/jira/browse/SOLR-17664
> > [2] https://issues.apache.org/jira/browse/SOLR-17620
> >
> > On Tue, Jul 22, 2025 at 1:12 AM David Smiley <ds...@apache.org> wrote:
> >
> > > On the epic of seeking the _eventual_ demise of the Overseer, I'm
> seeking
> > > to make it disabled for *new* SolrCloud clusters in Solr 10. --
> > > https://issues.apache.org/jira/browse/SOLR-17293
> > > The epic: https://issues.apache.org/jira/browse/SOLR-14927 (oddly no
> SIP
> > > but it has a doc anyway). I think it's sufficiently ready for the
> great
> > > majority of SolrCloud clusters. A cluster with a collection
> containing a
> > > thousand+ replicas might pose a performance concern on start/restart
> events
> > > due to independent replica state updates. Of course, with such a
> change,
> > > there will be a section in the upgrade page in the ref guide to advise
> > > users who may opt to make an explicit choice.
> > >
> > > I don't love that the new mode doesn't have an elegant/clear way to
> refer
> > > to it. The best I've come up with is to say what it *isn't* -- it
> *isn't*
> > > the Overseer. "The Overseer is disabled". Awkwardly there are two
> > > undocumented solr.xml booleans, both a mouthful:
> > > distributedClusterStateUpdates and
> > > distributedCollectionConfigSetExecution. I propose instead that a
> single
> > > boolean cluster property be defined named "overseer" or
> "overseerEnabled".
> > > FYI the existing known cluster properties are defined here:
> > > org.apache.solr.common.cloud.ZkStateReader#KNOWN_CLUSTER_PROPS
> > > Even if such a boolean is agreeable... it raises the question of what
> > > should become of the "overseer" node role.
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Jason Gerlowski <ge...@gmail.com>.
In the latest Virtual Community Meetup, Pierre and Bruno shared that
they recently enabled "no-overseer" mode (really need a better name!)
at their workplace in production and were hitting some stability
issues. Their anecdotes give me at least some "pause" about whether
this is really ready to be the default, or whether it needs a bit more
time to bake.
Any chance either of them are watching this thread and can provide an
update or their own 2c here?
Best,
Jason
On Wed, Aug 20, 2025 at 6:35 PM David Smiley <ds...@apache.org> wrote:
>
> Update: I proposed the Solr version go into cluster properties[1] on ZK
> initialization but Houston pushed back on that approach. The other
> approach is to rely on the "least Solr version" as returned by the Solr
> version being added to live nodes[2]. However that's very dynamic and I'm
> concerned about an old Solr version somehow joining an existing cluster.
> Rather than simply tell users "don't do that" (which we should do anyway),
> I'm inclined to have Solr fail to join an existing cluster if doing so
> would *lower* the least Solr version. Perhaps ignoring the final patch
> version. I plan on updating that PR accordingly. The PR would have to go
> into 9.x too, and thus such logic can't be enforced for older Solr
> versions. Regardless, an upgrading user can choose the setting.
>
> [1] https://issues.apache.org/jira/browse/SOLR-17664
> [2] https://issues.apache.org/jira/browse/SOLR-17620
>
> On Tue, Jul 22, 2025 at 1:12 AM David Smiley <ds...@apache.org> wrote:
>
> > On the epic of seeking the _eventual_ demise of the Overseer, I'm seeking
> > to make it disabled for *new* SolrCloud clusters in Solr 10. --
> > https://issues.apache.org/jira/browse/SOLR-17293
> > The epic: https://issues.apache.org/jira/browse/SOLR-14927 (oddly no SIP
> > but it has a doc anyway). I think it's sufficiently ready for the great
> > majority of SolrCloud clusters. A cluster with a collection containing a
> > thousand+ replicas might pose a performance concern on start/restart events
> > due to independent replica state updates. Of course, with such a change,
> > there will be a section in the upgrade page in the ref guide to advise
> > users who may opt to make an explicit choice.
> >
> > I don't love that the new mode doesn't have an elegant/clear way to refer
> > to it. The best I've come up with is to say what it *isn't* -- it *isn't*
> > the Overseer. "The Overseer is disabled". Awkwardly there are two
> > undocumented solr.xml booleans, both a mouthful:
> > distributedClusterStateUpdates and
> > distributedCollectionConfigSetExecution. I propose instead that a single
> > boolean cluster property be defined named "overseer" or "overseerEnabled".
> > FYI the existing known cluster properties are defined here:
> > org.apache.solr.common.cloud.ZkStateReader#KNOWN_CLUSTER_PROPS
> > Even if such a boolean is agreeable... it raises the question of what
> > should become of the "overseer" node role.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org