You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Ilan Ginzburg <il...@gmail.com> on 2025/10/01 13:59:54 UTC
Re: Propose: Disabled Overseer in Solr 10 as default
It's hard to provide a recommended threshold on collection size for
distributed mode.
I didn't run tests and it obviously depends on the number of nodes in the
cluster and how fast everything (including ZooKeeper) runs, but I'd say
that below a couple hundred replicas total for a collection it should be ok.
When a Solr node starts it marks all its replicas DOWN before marking them
ACTIVE. If PRS is not used this could take a long time with distributed
mode and be slower than Overseer due to the lack of batching of updates.
Indexing and query performance is obviously not impacted by distributed
mode or Overseer performance, unless shard split performance is considered
part of indexing performance.
Ilan
On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org> wrote:
> Let's imagine (quite realistically) that I'm updating our upgrade notes and
> providing advice for users to choose the Overseer or not. Given the
> benefits & risks, at what replica count threshold (for the biggest
> collection) would you advise continued use of the Overseer?
>
> Sufficient stability is the characteristic I'm looking for in the above
> question, not performance. I understand that it'd be ideal if the
> cluster's state was structured differently to improve performance of
> certain administrative operations, but performance is not a requirement for
> stability. The most important performance considerations our users have
> relate to index & query. There's a basic assumption that nodes can restart
> in a "reasonable time"... maybe you'd care to try to define that. I think
> your recommendations around restructuring the cluster state would
> ultimately impact the performance of restarts and some other administrative
> scenarios but shouldn't be a prerequisite for a stable system.
>
> On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> > Distributed mode doesn't behave nicely when there are many concurrent
> > updates to a given collection's state.json.
> >
> > I'd recommend *against* making it the default at this time.
> >
> > The "root cause" is the presence of replica specific information in
> > state.json. In addition to relatively rare cases of changes to the
> sharding
> > of the collection, state.json is updated when replicas are created or
> > destroyed or moved or have their properties changed, and when PRS is not
> > used, also when replicas change state (which happens a lot when a Solr
> node
> > restarts for example).
> >
> > Therefore before making distributed mode the default, something has to be
> > done.
> > As Pierre suggests, redesign Collection API operations that require
> > multiple updates to be more efficient and group them when executing in
> > distributed mode. Also make sure that smaller operations that happen
> > concurrently are efficient enough.
> > Another option is to remove replica information from state.json (keep
> > collection metadata and shard definitions there), and create state-
> > *<shardname>*.json for each shard with the replicas of that shard.
> > Contention on anything replica related will be restricted to replicas of
> > the same shard.
> > There will be more watches on ZooKeeper, they will trigger less often and
> > less data will be read each time. Also less data to compress/uncompress
> > each time state.json is written or read (when so configured).
> >
> > Throttling goes against making SolrCloud as fast as we can.
> >
> > SolrCloud started with a single clusterstate.json file describing all
> > collections (removed in 9.0), then moved to per collection state.json
> files
> > for scalability reasons.
> > Maybe the time has come to split that big blob further?
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> hossman_lucene@fucit.org
> > >
> > wrote:
> >
> > >
> > > : I don't think this should prevent shipping a system that is
> objectively
> > > way
> > > : simpler than the Overseer. Solr 10 will have both modes, no matter
> > what
> > > : the default is. Changing the default makes it easier to remove it in
> > > Solr
> > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > amazing!
> > >
> > > I'm not understanding yoru claim that changing a default from A(x) to
> > A(y)
> > > in 10.0 makes removing A(x) in 11.0 easier?
> > >
> > > You could change the default in 10.1, 10.2, etc... and it would still
> be
> > > the same amount of effort to remove it in 11.0.
> > >
> > > No matter when you change the default, if the *option* to use A(x)
> still
> > > exists in all versions < 11.0, then any "removal" of the code
> > implementing
> > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> > > code/process/documentation enabling users to migrate their cluster to
> > > A(y)
> > >
> > >
> > > -Hoss
> > > http://www.lucidworks.com/
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Anshum Gupta <an...@anshumgupta.net>.
I agree, I'm worried about the same i.e. defaulting to a mode that comes
with caveats and isn't really tested and hardened.
On Wed, Oct 1, 2025 at 9:35 AM Jason Gerlowski <ge...@gmail.com>
wrote:
> > Let's imagine (quite realistically) that I'm updating our upgrade notes
> and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
>
> Even assuming we can figure out a value on 'N' that sounds reasonable,
> can be shown to be stable in load testing, etc....is that enough to
> recommend "non-overseer" mode?
>
> Sizing (which includes replica-count) is notoriously a guess-and-check
> process in Solr. And even for users who have done everything right
> and dialed in their replica-count with some benchmarking - what
> happens when their requirements change and they need to add replicas
> to (e.g.) accommodate a higher QPS. Is there an easy way for those
> users to switch back to the overseer, or do they have to risk
> instability going forward?
>
> I guess I'm worried about basing recommendations on a factor like
> replica-count which has a tendency to drift over time, if the decision
> itself (i.e. overseer or not) is difficult to reverse after the fact.
> Not 100% sure that's the case here, but that's my suspicion based on a
> hazy recollection of some past discussions.
>
> Best,
>
> Jason
>
> On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > It's hard to provide a recommended threshold on collection size for
> > distributed mode.
> > I didn't run tests and it obviously depends on the number of nodes in the
> > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > that below a couple hundred replicas total for a collection it should be
> ok.
> > When a Solr node starts it marks all its replicas DOWN before marking
> them
> > ACTIVE. If PRS is not used this could take a long time with distributed
> > mode and be slower than Overseer due to the lack of batching of updates.
> >
> > Indexing and query performance is obviously not impacted by distributed
> > mode or Overseer performance, unless shard split performance is
> considered
> > part of indexing performance.
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> > >
> > > Sufficient stability is the characteristic I'm looking for in the above
> > > question, not performance. I understand that it'd be ideal if the
> > > cluster's state was structured differently to improve performance of
> > > certain administrative operations, but performance is not a
> requirement for
> > > stability. The most important performance considerations our users
> have
> > > relate to index & query. There's a basic assumption that nodes can
> restart
> > > in a "reasonable time"... maybe you'd care to try to define that. I
> think
> > > your recommendations around restructuring the cluster state would
> > > ultimately impact the performance of restarts and some other
> administrative
> > > scenarios but shouldn't be a prerequisite for a stable system.
> > >
> > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > > Distributed mode doesn't behave nicely when there are many concurrent
> > > > updates to a given collection's state.json.
> > > >
> > > > I'd recommend *against* making it the default at this time.
> > > >
> > > > The "root cause" is the presence of replica specific information in
> > > > state.json. In addition to relatively rare cases of changes to the
> > > sharding
> > > > of the collection, state.json is updated when replicas are created or
> > > > destroyed or moved or have their properties changed, and when PRS is
> not
> > > > used, also when replicas change state (which happens a lot when a
> Solr
> > > node
> > > > restarts for example).
> > > >
> > > > Therefore before making distributed mode the default, something has
> to be
> > > > done.
> > > > As Pierre suggests, redesign Collection API operations that require
> > > > multiple updates to be more efficient and group them when executing
> in
> > > > distributed mode. Also make sure that smaller operations that happen
> > > > concurrently are efficient enough.
> > > > Another option is to remove replica information from state.json (keep
> > > > collection metadata and shard definitions there), and create state-
> > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > Contention on anything replica related will be restricted to
> replicas of
> > > > the same shard.
> > > > There will be more watches on ZooKeeper, they will trigger less
> often and
> > > > less data will be read each time. Also less data to
> compress/uncompress
> > > > each time state.json is written or read (when so configured).
> > > >
> > > > Throttling goes against making SolrCloud as fast as we can.
> > > >
> > > > SolrCloud started with a single clusterstate.json file describing all
> > > > collections (removed in 9.0), then moved to per collection state.json
> > > files
> > > > for scalability reasons.
> > > > Maybe the time has come to split that big blob further?
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > hossman_lucene@fucit.org
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > : I don't think this should prevent shipping a system that is
> > > objectively
> > > > > way
> > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> matter
> > > > what
> > > > > : the default is. Changing the default makes it easier to remove
> it in
> > > > > Solr
> > > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > > amazing!
> > > > >
> > > > > I'm not understanding yoru claim that changing a default from A(x)
> to
> > > > A(y)
> > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > >
> > > > > You could change the default in 10.1, 10.2, etc... and it would
> still
> > > be
> > > > > the same amount of effort to remove it in 11.0.
> > > > >
> > > > > No matter when you change the default, if the *option* to use A(x)
> > > still
> > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > implementing
> > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> some
> > > > > code/process/documentation enabling users to migrate their cluster
> to
> > > > > A(y)
> > > > >
> > > > >
> > > > > -Hoss
> > > > > http://www.lucidworks.com/
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
Anshum Gupta
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by David Smiley <ds...@apache.org>.
Anshum, it's been "tested" since February 2021, so we're talking 4.5 years
now:
https://github.com/apache/solr/commit/23755ddfdd36a9613010cb9e6201127df55be744#diff-e49e335c95e0b12e7beda444cf67542457800de47dae39034af46a6797327776R125
(half of most SolrCloud test runs).
"Hardening" (say real world users, not tests): There is a chicken & egg
problem here, as it's opt-in. I'm glad Salesforce *finally* (after having
developed the thing! Jeesh) flipped the switch and shared some feedback
pertinent to large collections, albeit not pertinent to small or medium
ones (IMO). For many users, the Overseer is effectively an invisible
internal detail, so flipping this isn't going to be on people's radar. It
certainly wasn't in the past because the feature was extremely invisible /
undocumented. I recently (frantically?) worked on addressing that concern,
and giving it a name (overseer disabled). Even if users never care about
this (and why should they TBH), as a maintainer of SolrCloud, I care about
this very much! And really, you all should too. I can't think of any
positive improvement to SolrCloud's *maintainability* with a bigger impact
than removing the Overseer. Removing it is a gift to ourselves, and to any
user/operator of a cluster who has the misfortune of debugging Overseer
interactions.
I was pondering keeping both modes active in a cluster on a per-collection
basis (like PRS can do) which doesn't sound hard... but perhaps any time
spent on that would be better spent on improving the instability. If there
was a performance test designed to expose the instability, it'd do wonders
to prove out relatively cheap work-arounds like I suggested. Re-designing
cluster state, on the other hand, is a noble but costly endeavor. We'd
still need a test _anyway_ to know if the instability is fixed, or a brave
user :-)
On Wed, Oct 1, 2025 at 7:03 PM Ilan Ginzburg <il...@gmail.com> wrote:
> The design doc:
>
> https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit
>
> Timothy Potter identifying the issue right away:
>
> https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit?disco=AAAAHGxplOw
>
>
>
>
> On Thu, Oct 2, 2025 at 12:09 AM Ishan Chattopadhyaya
> <ic...@gmail.com> wrote:
> >
> > Is there a SIP or design document for this feature (no overseer mode),
> > please?
> > I'm familiar with the traditional overseer based mode, the PRS mode, not
> > very familiar with the DSU thing that Ilan + colleagues introduced. But,
> I
> > can't find enough design notes for the no overseer mode.
> >
> > On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
> >
> > > Sounds like an experimental feature that could end up changing (to fix
> this
> > > issue with replica count). While we want feedback from real users, this
> > > discussion makes me think it's just not ready for default status yet.
> > >
> > > On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <gerlowskija@gmail.com
> >
> > > wrote:
> > >
> > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes
> > > > and
> > > > > providing advice for users to choose the Overseer or not. Given
> the
> > > > > benefits & risks, at what replica count threshold (for the biggest
> > > > > collection) would you advise continued use of the Overseer?
> > > >
> > > > Even assuming we can figure out a value on 'N' that sounds
> reasonable,
> > > > can be shown to be stable in load testing, etc....is that enough to
> > > > recommend "non-overseer" mode?
> > > >
> > > > Sizing (which includes replica-count) is notoriously a
> guess-and-check
> > > > process in Solr. And even for users who have done everything right
> > > > and dialed in their replica-count with some benchmarking - what
> > > > happens when their requirements change and they need to add replicas
> > > > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > > > users to switch back to the overseer, or do they have to risk
> > > > instability going forward?
> > > >
> > > > I guess I'm worried about basing recommendations on a factor like
> > > > replica-count which has a tendency to drift over time, if the
> decision
> > > > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > > > Not 100% sure that's the case here, but that's my suspicion based on
> a
> > > > hazy recollection of some past discussions.
> > > >
> > > > Best,
> > > >
> > > > Jason
> > > >
> > > > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> > > wrote:
> > > > >
> > > > > It's hard to provide a recommended threshold on collection size for
> > > > > distributed mode.
> > > > > I didn't run tests and it obviously depends on the number of nodes
> in
> > > the
> > > > > cluster and how fast everything (including ZooKeeper) runs, but
> I'd say
> > > > > that below a couple hundred replicas total for a collection it
> should
> > > be
> > > > ok.
> > > > > When a Solr node starts it marks all its replicas DOWN before
> marking
> > > > them
> > > > > ACTIVE. If PRS is not used this could take a long time with
> distributed
> > > > > mode and be slower than Overseer due to the lack of batching of
> > > updates.
> > > > >
> > > > > Indexing and query performance is obviously not impacted by
> distributed
> > > > > mode or Overseer performance, unless shard split performance is
> > > > considered
> > > > > part of indexing performance.
> > > > >
> > > > > Ilan
> > > > >
> > > > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > > > notes and
> > > > > > providing advice for users to choose the Overseer or not. Given
> the
> > > > > > benefits & risks, at what replica count threshold (for the
> biggest
> > > > > > collection) would you advise continued use of the Overseer?
> > > > > >
> > > > > > Sufficient stability is the characteristic I'm looking for in the
> > > above
> > > > > > question, not performance. I understand that it'd be ideal if
> the
> > > > > > cluster's state was structured differently to improve
> performance of
> > > > > > certain administrative operations, but performance is not a
> > > > requirement for
> > > > > > stability. The most important performance considerations our
> users
> > > > have
> > > > > > relate to index & query. There's a basic assumption that nodes
> can
> > > > restart
> > > > > > in a "reasonable time"... maybe you'd care to try to define
> that. I
> > > > think
> > > > > > your recommendations around restructuring the cluster state would
> > > > > > ultimately impact the performance of restarts and some other
> > > > administrative
> > > > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > > > >
> > > > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <
> ilansolr@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Distributed mode doesn't behave nicely when there are many
> > > concurrent
> > > > > > > updates to a given collection's state.json.
> > > > > > >
> > > > > > > I'd recommend *against* making it the default at this time.
> > > > > > >
> > > > > > > The "root cause" is the presence of replica specific
> information in
> > > > > > > state.json. In addition to relatively rare cases of changes to
> the
> > > > > > sharding
> > > > > > > of the collection, state.json is updated when replicas are
> created
> > > or
> > > > > > > destroyed or moved or have their properties changed, and when
> PRS
> > > is
> > > > not
> > > > > > > used, also when replicas change state (which happens a lot
> when a
> > > > Solr
> > > > > > node
> > > > > > > restarts for example).
> > > > > > >
> > > > > > > Therefore before making distributed mode the default,
> something has
> > > > to be
> > > > > > > done.
> > > > > > > As Pierre suggests, redesign Collection API operations that
> require
> > > > > > > multiple updates to be more efficient and group them when
> executing
> > > > in
> > > > > > > distributed mode. Also make sure that smaller operations that
> > > happen
> > > > > > > concurrently are efficient enough.
> > > > > > > Another option is to remove replica information from state.json
> > > (keep
> > > > > > > collection metadata and shard definitions there), and create
> state-
> > > > > > > *<shardname>*.json for each shard with the replicas of that
> shard.
> > > > > > > Contention on anything replica related will be restricted to
> > > > replicas of
> > > > > > > the same shard.
> > > > > > > There will be more watches on ZooKeeper, they will trigger less
> > > > often and
> > > > > > > less data will be read each time. Also less data to
> > > > compress/uncompress
> > > > > > > each time state.json is written or read (when so configured).
> > > > > > >
> > > > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > > > >
> > > > > > > SolrCloud started with a single clusterstate.json file
> describing
> > > all
> > > > > > > collections (removed in 9.0), then moved to per collection
> > > state.json
> > > > > > files
> > > > > > > for scalability reasons.
> > > > > > > Maybe the time has come to split that big blob further?
> > > > > > >
> > > > > > > Ilan
> > > > > > >
> > > > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > > > hossman_lucene@fucit.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > : I don't think this should prevent shipping a system that is
> > > > > > objectively
> > > > > > > > way
> > > > > > > > : simpler than the Overseer. Solr 10 will have both modes,
> no
> > > > matter
> > > > > > > what
> > > > > > > > : the default is. Changing the default makes it easier to
> remove
> > > > it in
> > > > > > > > Solr
> > > > > > > > : 11. The impact on ease of understanding SolrCloud in 11
> will
> > > be
> > > > > > > amazing!
> > > > > > > >
> > > > > > > > I'm not understanding yoru claim that changing a default from
> > > A(x)
> > > > to
> > > > > > > A(y)
> > > > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > > > >
> > > > > > > > You could change the default in 10.1, 10.2, etc... and it
> would
> > > > still
> > > > > > be
> > > > > > > > the same amount of effort to remove it in 11.0.
> > > > > > > >
> > > > > > > > No matter when you change the default, if the *option* to use
> > > A(x)
> > > > > > still
> > > > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > > > implementing
> > > > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0
> have
> > > > some
> > > > > > > > code/process/documentation enabling users to migrate their
> > > cluster
> > > > to
> > > > > > > > A(y)
> > > > > > > >
> > > > > > > >
> > > > > > > > -Hoss
> > > > > > > > http://www.lucidworks.com/
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> > > --
> > > http://www.needhamsoftware.com (work)
> > > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ilan Ginzburg <il...@gmail.com>.
The design doc:
https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit
Timothy Potter identifying the issue right away:
https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit?disco=AAAAHGxplOw
On Thu, Oct 2, 2025 at 12:09 AM Ishan Chattopadhyaya
<ic...@gmail.com> wrote:
>
> Is there a SIP or design document for this feature (no overseer mode),
> please?
> I'm familiar with the traditional overseer based mode, the PRS mode, not
> very familiar with the DSU thing that Ilan + colleagues introduced. But, I
> can't find enough design notes for the no overseer mode.
>
> On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
>
> > Sounds like an experimental feature that could end up changing (to fix this
> > issue with replica count). While we want feedback from real users, this
> > discussion makes me think it's just not ready for default status yet.
> >
> > On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
> > wrote:
> >
> > > > Let's imagine (quite realistically) that I'm updating our upgrade notes
> > > and
> > > > providing advice for users to choose the Overseer or not. Given the
> > > > benefits & risks, at what replica count threshold (for the biggest
> > > > collection) would you advise continued use of the Overseer?
> > >
> > > Even assuming we can figure out a value on 'N' that sounds reasonable,
> > > can be shown to be stable in load testing, etc....is that enough to
> > > recommend "non-overseer" mode?
> > >
> > > Sizing (which includes replica-count) is notoriously a guess-and-check
> > > process in Solr. And even for users who have done everything right
> > > and dialed in their replica-count with some benchmarking - what
> > > happens when their requirements change and they need to add replicas
> > > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > > users to switch back to the overseer, or do they have to risk
> > > instability going forward?
> > >
> > > I guess I'm worried about basing recommendations on a factor like
> > > replica-count which has a tendency to drift over time, if the decision
> > > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > > Not 100% sure that's the case here, but that's my suspicion based on a
> > > hazy recollection of some past discussions.
> > >
> > > Best,
> > >
> > > Jason
> > >
> > > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> > wrote:
> > > >
> > > > It's hard to provide a recommended threshold on collection size for
> > > > distributed mode.
> > > > I didn't run tests and it obviously depends on the number of nodes in
> > the
> > > > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > > > that below a couple hundred replicas total for a collection it should
> > be
> > > ok.
> > > > When a Solr node starts it marks all its replicas DOWN before marking
> > > them
> > > > ACTIVE. If PRS is not used this could take a long time with distributed
> > > > mode and be slower than Overseer due to the lack of batching of
> > updates.
> > > >
> > > > Indexing and query performance is obviously not impacted by distributed
> > > > mode or Overseer performance, unless shard split performance is
> > > considered
> > > > part of indexing performance.
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > > wrote:
> > > >
> > > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > > notes and
> > > > > providing advice for users to choose the Overseer or not. Given the
> > > > > benefits & risks, at what replica count threshold (for the biggest
> > > > > collection) would you advise continued use of the Overseer?
> > > > >
> > > > > Sufficient stability is the characteristic I'm looking for in the
> > above
> > > > > question, not performance. I understand that it'd be ideal if the
> > > > > cluster's state was structured differently to improve performance of
> > > > > certain administrative operations, but performance is not a
> > > requirement for
> > > > > stability. The most important performance considerations our users
> > > have
> > > > > relate to index & query. There's a basic assumption that nodes can
> > > restart
> > > > > in a "reasonable time"... maybe you'd care to try to define that. I
> > > think
> > > > > your recommendations around restructuring the cluster state would
> > > > > ultimately impact the performance of restarts and some other
> > > administrative
> > > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > > >
> > > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Distributed mode doesn't behave nicely when there are many
> > concurrent
> > > > > > updates to a given collection's state.json.
> > > > > >
> > > > > > I'd recommend *against* making it the default at this time.
> > > > > >
> > > > > > The "root cause" is the presence of replica specific information in
> > > > > > state.json. In addition to relatively rare cases of changes to the
> > > > > sharding
> > > > > > of the collection, state.json is updated when replicas are created
> > or
> > > > > > destroyed or moved or have their properties changed, and when PRS
> > is
> > > not
> > > > > > used, also when replicas change state (which happens a lot when a
> > > Solr
> > > > > node
> > > > > > restarts for example).
> > > > > >
> > > > > > Therefore before making distributed mode the default, something has
> > > to be
> > > > > > done.
> > > > > > As Pierre suggests, redesign Collection API operations that require
> > > > > > multiple updates to be more efficient and group them when executing
> > > in
> > > > > > distributed mode. Also make sure that smaller operations that
> > happen
> > > > > > concurrently are efficient enough.
> > > > > > Another option is to remove replica information from state.json
> > (keep
> > > > > > collection metadata and shard definitions there), and create state-
> > > > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > > > Contention on anything replica related will be restricted to
> > > replicas of
> > > > > > the same shard.
> > > > > > There will be more watches on ZooKeeper, they will trigger less
> > > often and
> > > > > > less data will be read each time. Also less data to
> > > compress/uncompress
> > > > > > each time state.json is written or read (when so configured).
> > > > > >
> > > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > > >
> > > > > > SolrCloud started with a single clusterstate.json file describing
> > all
> > > > > > collections (removed in 9.0), then moved to per collection
> > state.json
> > > > > files
> > > > > > for scalability reasons.
> > > > > > Maybe the time has come to split that big blob further?
> > > > > >
> > > > > > Ilan
> > > > > >
> > > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > > hossman_lucene@fucit.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > : I don't think this should prevent shipping a system that is
> > > > > objectively
> > > > > > > way
> > > > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> > > matter
> > > > > > what
> > > > > > > : the default is. Changing the default makes it easier to remove
> > > it in
> > > > > > > Solr
> > > > > > > : 11. The impact on ease of understanding SolrCloud in 11 will
> > be
> > > > > > amazing!
> > > > > > >
> > > > > > > I'm not understanding yoru claim that changing a default from
> > A(x)
> > > to
> > > > > > A(y)
> > > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > > >
> > > > > > > You could change the default in 10.1, 10.2, etc... and it would
> > > still
> > > > > be
> > > > > > > the same amount of effort to remove it in 11.0.
> > > > > > >
> > > > > > > No matter when you change the default, if the *option* to use
> > A(x)
> > > > > still
> > > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > > implementing
> > > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> > > some
> > > > > > > code/process/documentation enabling users to migrate their
> > cluster
> > > to
> > > > > > > A(y)
> > > > > > >
> > > > > > >
> > > > > > > -Hoss
> > > > > > > http://www.lucidworks.com/
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Is there a SIP or design document for this feature (no overseer mode),
please?
I'm familiar with the traditional overseer based mode, the PRS mode, not
very familiar with the DSU thing that Ilan + colleagues introduced. But, I
can't find enough design notes for the no overseer mode.
On Wed, 1 Oct 2025 at 22:54, Gus Heck <gu...@gmail.com> wrote:
> Sounds like an experimental feature that could end up changing (to fix this
> issue with replica count). While we want feedback from real users, this
> discussion makes me think it's just not ready for default status yet.
>
> On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
> > > Let's imagine (quite realistically) that I'm updating our upgrade notes
> > and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> >
> > Even assuming we can figure out a value on 'N' that sounds reasonable,
> > can be shown to be stable in load testing, etc....is that enough to
> > recommend "non-overseer" mode?
> >
> > Sizing (which includes replica-count) is notoriously a guess-and-check
> > process in Solr. And even for users who have done everything right
> > and dialed in their replica-count with some benchmarking - what
> > happens when their requirements change and they need to add replicas
> > to (e.g.) accommodate a higher QPS. Is there an easy way for those
> > users to switch back to the overseer, or do they have to risk
> > instability going forward?
> >
> > I guess I'm worried about basing recommendations on a factor like
> > replica-count which has a tendency to drift over time, if the decision
> > itself (i.e. overseer or not) is difficult to reverse after the fact.
> > Not 100% sure that's the case here, but that's my suspicion based on a
> > hazy recollection of some past discussions.
> >
> > Best,
> >
> > Jason
> >
> > On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > It's hard to provide a recommended threshold on collection size for
> > > distributed mode.
> > > I didn't run tests and it obviously depends on the number of nodes in
> the
> > > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > > that below a couple hundred replicas total for a collection it should
> be
> > ok.
> > > When a Solr node starts it marks all its replicas DOWN before marking
> > them
> > > ACTIVE. If PRS is not used this could take a long time with distributed
> > > mode and be slower than Overseer due to the lack of batching of
> updates.
> > >
> > > Indexing and query performance is obviously not impacted by distributed
> > > mode or Overseer performance, unless shard split performance is
> > considered
> > > part of indexing performance.
> > >
> > > Ilan
> > >
> > > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> > wrote:
> > >
> > > > Let's imagine (quite realistically) that I'm updating our upgrade
> > notes and
> > > > providing advice for users to choose the Overseer or not. Given the
> > > > benefits & risks, at what replica count threshold (for the biggest
> > > > collection) would you advise continued use of the Overseer?
> > > >
> > > > Sufficient stability is the characteristic I'm looking for in the
> above
> > > > question, not performance. I understand that it'd be ideal if the
> > > > cluster's state was structured differently to improve performance of
> > > > certain administrative operations, but performance is not a
> > requirement for
> > > > stability. The most important performance considerations our users
> > have
> > > > relate to index & query. There's a basic assumption that nodes can
> > restart
> > > > in a "reasonable time"... maybe you'd care to try to define that. I
> > think
> > > > your recommendations around restructuring the cluster state would
> > > > ultimately impact the performance of restarts and some other
> > administrative
> > > > scenarios but shouldn't be a prerequisite for a stable system.
> > > >
> > > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> > wrote:
> > > >
> > > > > Distributed mode doesn't behave nicely when there are many
> concurrent
> > > > > updates to a given collection's state.json.
> > > > >
> > > > > I'd recommend *against* making it the default at this time.
> > > > >
> > > > > The "root cause" is the presence of replica specific information in
> > > > > state.json. In addition to relatively rare cases of changes to the
> > > > sharding
> > > > > of the collection, state.json is updated when replicas are created
> or
> > > > > destroyed or moved or have their properties changed, and when PRS
> is
> > not
> > > > > used, also when replicas change state (which happens a lot when a
> > Solr
> > > > node
> > > > > restarts for example).
> > > > >
> > > > > Therefore before making distributed mode the default, something has
> > to be
> > > > > done.
> > > > > As Pierre suggests, redesign Collection API operations that require
> > > > > multiple updates to be more efficient and group them when executing
> > in
> > > > > distributed mode. Also make sure that smaller operations that
> happen
> > > > > concurrently are efficient enough.
> > > > > Another option is to remove replica information from state.json
> (keep
> > > > > collection metadata and shard definitions there), and create state-
> > > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > > Contention on anything replica related will be restricted to
> > replicas of
> > > > > the same shard.
> > > > > There will be more watches on ZooKeeper, they will trigger less
> > often and
> > > > > less data will be read each time. Also less data to
> > compress/uncompress
> > > > > each time state.json is written or read (when so configured).
> > > > >
> > > > > Throttling goes against making SolrCloud as fast as we can.
> > > > >
> > > > > SolrCloud started with a single clusterstate.json file describing
> all
> > > > > collections (removed in 9.0), then moved to per collection
> state.json
> > > > files
> > > > > for scalability reasons.
> > > > > Maybe the time has come to split that big blob further?
> > > > >
> > > > > Ilan
> > > > >
> > > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > > hossman_lucene@fucit.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > : I don't think this should prevent shipping a system that is
> > > > objectively
> > > > > > way
> > > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> > matter
> > > > > what
> > > > > > : the default is. Changing the default makes it easier to remove
> > it in
> > > > > > Solr
> > > > > > : 11. The impact on ease of understanding SolrCloud in 11 will
> be
> > > > > amazing!
> > > > > >
> > > > > > I'm not understanding yoru claim that changing a default from
> A(x)
> > to
> > > > > A(y)
> > > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > > >
> > > > > > You could change the default in 10.1, 10.2, etc... and it would
> > still
> > > > be
> > > > > > the same amount of effort to remove it in 11.0.
> > > > > >
> > > > > > No matter when you change the default, if the *option* to use
> A(x)
> > > > still
> > > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > > implementing
> > > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> > some
> > > > > > code/process/documentation enabling users to migrate their
> cluster
> > to
> > > > > > A(y)
> > > > > >
> > > > > >
> > > > > > -Hoss
> > > > > > http://www.lucidworks.com/
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Gus Heck <gu...@gmail.com>.
Sounds like an experimental feature that could end up changing (to fix this
issue with replica count). While we want feedback from real users, this
discussion makes me think it's just not ready for default status yet.
On Wed, Oct 1, 2025 at 12:44 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> > Let's imagine (quite realistically) that I'm updating our upgrade notes
> and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
>
> Even assuming we can figure out a value on 'N' that sounds reasonable,
> can be shown to be stable in load testing, etc....is that enough to
> recommend "non-overseer" mode?
>
> Sizing (which includes replica-count) is notoriously a guess-and-check
> process in Solr. And even for users who have done everything right
> and dialed in their replica-count with some benchmarking - what
> happens when their requirements change and they need to add replicas
> to (e.g.) accommodate a higher QPS. Is there an easy way for those
> users to switch back to the overseer, or do they have to risk
> instability going forward?
>
> I guess I'm worried about basing recommendations on a factor like
> replica-count which has a tendency to drift over time, if the decision
> itself (i.e. overseer or not) is difficult to reverse after the fact.
> Not 100% sure that's the case here, but that's my suspicion based on a
> hazy recollection of some past discussions.
>
> Best,
>
> Jason
>
> On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > It's hard to provide a recommended threshold on collection size for
> > distributed mode.
> > I didn't run tests and it obviously depends on the number of nodes in the
> > cluster and how fast everything (including ZooKeeper) runs, but I'd say
> > that below a couple hundred replicas total for a collection it should be
> ok.
> > When a Solr node starts it marks all its replicas DOWN before marking
> them
> > ACTIVE. If PRS is not used this could take a long time with distributed
> > mode and be slower than Overseer due to the lack of batching of updates.
> >
> > Indexing and query performance is obviously not impacted by distributed
> > mode or Overseer performance, unless shard split performance is
> considered
> > part of indexing performance.
> >
> > Ilan
> >
> > On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > Let's imagine (quite realistically) that I'm updating our upgrade
> notes and
> > > providing advice for users to choose the Overseer or not. Given the
> > > benefits & risks, at what replica count threshold (for the biggest
> > > collection) would you advise continued use of the Overseer?
> > >
> > > Sufficient stability is the characteristic I'm looking for in the above
> > > question, not performance. I understand that it'd be ideal if the
> > > cluster's state was structured differently to improve performance of
> > > certain administrative operations, but performance is not a
> requirement for
> > > stability. The most important performance considerations our users
> have
> > > relate to index & query. There's a basic assumption that nodes can
> restart
> > > in a "reasonable time"... maybe you'd care to try to define that. I
> think
> > > your recommendations around restructuring the cluster state would
> > > ultimately impact the performance of restarts and some other
> administrative
> > > scenarios but shouldn't be a prerequisite for a stable system.
> > >
> > > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > > Distributed mode doesn't behave nicely when there are many concurrent
> > > > updates to a given collection's state.json.
> > > >
> > > > I'd recommend *against* making it the default at this time.
> > > >
> > > > The "root cause" is the presence of replica specific information in
> > > > state.json. In addition to relatively rare cases of changes to the
> > > sharding
> > > > of the collection, state.json is updated when replicas are created or
> > > > destroyed or moved or have their properties changed, and when PRS is
> not
> > > > used, also when replicas change state (which happens a lot when a
> Solr
> > > node
> > > > restarts for example).
> > > >
> > > > Therefore before making distributed mode the default, something has
> to be
> > > > done.
> > > > As Pierre suggests, redesign Collection API operations that require
> > > > multiple updates to be more efficient and group them when executing
> in
> > > > distributed mode. Also make sure that smaller operations that happen
> > > > concurrently are efficient enough.
> > > > Another option is to remove replica information from state.json (keep
> > > > collection metadata and shard definitions there), and create state-
> > > > *<shardname>*.json for each shard with the replicas of that shard.
> > > > Contention on anything replica related will be restricted to
> replicas of
> > > > the same shard.
> > > > There will be more watches on ZooKeeper, they will trigger less
> often and
> > > > less data will be read each time. Also less data to
> compress/uncompress
> > > > each time state.json is written or read (when so configured).
> > > >
> > > > Throttling goes against making SolrCloud as fast as we can.
> > > >
> > > > SolrCloud started with a single clusterstate.json file describing all
> > > > collections (removed in 9.0), then moved to per collection state.json
> > > files
> > > > for scalability reasons.
> > > > Maybe the time has come to split that big blob further?
> > > >
> > > > Ilan
> > > >
> > > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > > hossman_lucene@fucit.org
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > : I don't think this should prevent shipping a system that is
> > > objectively
> > > > > way
> > > > > : simpler than the Overseer. Solr 10 will have both modes, no
> matter
> > > > what
> > > > > : the default is. Changing the default makes it easier to remove
> it in
> > > > > Solr
> > > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > > amazing!
> > > > >
> > > > > I'm not understanding yoru claim that changing a default from A(x)
> to
> > > > A(y)
> > > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > > >
> > > > > You could change the default in 10.1, 10.2, etc... and it would
> still
> > > be
> > > > > the same amount of effort to remove it in 11.0.
> > > > >
> > > > > No matter when you change the default, if the *option* to use A(x)
> > > still
> > > > > exists in all versions < 11.0, then any "removal" of the code
> > > > implementing
> > > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have
> some
> > > > > code/process/documentation enabling users to migrate their cluster
> to
> > > > > A(y)
> > > > >
> > > > >
> > > > > -Hoss
> > > > > http://www.lucidworks.com/
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)
Re: Propose: Disabled Overseer in Solr 10 as default
Posted by Jason Gerlowski <ge...@gmail.com>.
> Let's imagine (quite realistically) that I'm updating our upgrade notes and
> providing advice for users to choose the Overseer or not. Given the
> benefits & risks, at what replica count threshold (for the biggest
> collection) would you advise continued use of the Overseer?
Even assuming we can figure out a value on 'N' that sounds reasonable,
can be shown to be stable in load testing, etc....is that enough to
recommend "non-overseer" mode?
Sizing (which includes replica-count) is notoriously a guess-and-check
process in Solr. And even for users who have done everything right
and dialed in their replica-count with some benchmarking - what
happens when their requirements change and they need to add replicas
to (e.g.) accommodate a higher QPS. Is there an easy way for those
users to switch back to the overseer, or do they have to risk
instability going forward?
I guess I'm worried about basing recommendations on a factor like
replica-count which has a tendency to drift over time, if the decision
itself (i.e. overseer or not) is difficult to reverse after the fact.
Not 100% sure that's the case here, but that's my suspicion based on a
hazy recollection of some past discussions.
Best,
Jason
On Wed, Oct 1, 2025 at 10:10 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> It's hard to provide a recommended threshold on collection size for
> distributed mode.
> I didn't run tests and it obviously depends on the number of nodes in the
> cluster and how fast everything (including ZooKeeper) runs, but I'd say
> that below a couple hundred replicas total for a collection it should be ok.
> When a Solr node starts it marks all its replicas DOWN before marking them
> ACTIVE. If PRS is not used this could take a long time with distributed
> mode and be slower than Overseer due to the lack of batching of updates.
>
> Indexing and query performance is obviously not impacted by distributed
> mode or Overseer performance, unless shard split performance is considered
> part of indexing performance.
>
> Ilan
>
> On Tue, Sep 30, 2025 at 10:20 PM David Smiley <ds...@apache.org> wrote:
>
> > Let's imagine (quite realistically) that I'm updating our upgrade notes and
> > providing advice for users to choose the Overseer or not. Given the
> > benefits & risks, at what replica count threshold (for the biggest
> > collection) would you advise continued use of the Overseer?
> >
> > Sufficient stability is the characteristic I'm looking for in the above
> > question, not performance. I understand that it'd be ideal if the
> > cluster's state was structured differently to improve performance of
> > certain administrative operations, but performance is not a requirement for
> > stability. The most important performance considerations our users have
> > relate to index & query. There's a basic assumption that nodes can restart
> > in a "reasonable time"... maybe you'd care to try to define that. I think
> > your recommendations around restructuring the cluster state would
> > ultimately impact the performance of restarts and some other administrative
> > scenarios but shouldn't be a prerequisite for a stable system.
> >
> > On Tue, Sep 30, 2025 at 4:20 AM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > > Distributed mode doesn't behave nicely when there are many concurrent
> > > updates to a given collection's state.json.
> > >
> > > I'd recommend *against* making it the default at this time.
> > >
> > > The "root cause" is the presence of replica specific information in
> > > state.json. In addition to relatively rare cases of changes to the
> > sharding
> > > of the collection, state.json is updated when replicas are created or
> > > destroyed or moved or have their properties changed, and when PRS is not
> > > used, also when replicas change state (which happens a lot when a Solr
> > node
> > > restarts for example).
> > >
> > > Therefore before making distributed mode the default, something has to be
> > > done.
> > > As Pierre suggests, redesign Collection API operations that require
> > > multiple updates to be more efficient and group them when executing in
> > > distributed mode. Also make sure that smaller operations that happen
> > > concurrently are efficient enough.
> > > Another option is to remove replica information from state.json (keep
> > > collection metadata and shard definitions there), and create state-
> > > *<shardname>*.json for each shard with the replicas of that shard.
> > > Contention on anything replica related will be restricted to replicas of
> > > the same shard.
> > > There will be more watches on ZooKeeper, they will trigger less often and
> > > less data will be read each time. Also less data to compress/uncompress
> > > each time state.json is written or read (when so configured).
> > >
> > > Throttling goes against making SolrCloud as fast as we can.
> > >
> > > SolrCloud started with a single clusterstate.json file describing all
> > > collections (removed in 9.0), then moved to per collection state.json
> > files
> > > for scalability reasons.
> > > Maybe the time has come to split that big blob further?
> > >
> > > Ilan
> > >
> > > On Tue, Sep 30, 2025 at 12:40 AM Chris Hostetter <
> > hossman_lucene@fucit.org
> > > >
> > > wrote:
> > >
> > > >
> > > > : I don't think this should prevent shipping a system that is
> > objectively
> > > > way
> > > > : simpler than the Overseer. Solr 10 will have both modes, no matter
> > > what
> > > > : the default is. Changing the default makes it easier to remove it in
> > > > Solr
> > > > : 11. The impact on ease of understanding SolrCloud in 11 will be
> > > amazing!
> > > >
> > > > I'm not understanding yoru claim that changing a default from A(x) to
> > > A(y)
> > > > in 10.0 makes removing A(x) in 11.0 easier?
> > > >
> > > > You could change the default in 10.1, 10.2, etc... and it would still
> > be
> > > > the same amount of effort to remove it in 11.0.
> > > >
> > > > No matter when you change the default, if the *option* to use A(x)
> > still
> > > > exists in all versions < 11.0, then any "removal" of the code
> > > implementing
> > > > A(x) in 11.0 still needs to ensure that all versions >= 11.0 have some
> > > > code/process/documentation enabling users to migrate their cluster to
> > > > A(y)
> > > >
> > > >
> > > > -Hoss
> > > > http://www.lucidworks.com/
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org