Change suggestion: more efficient replica state tracking

[Ilan Ginzburg <il...@gmail.com>]

*TL;DR; a way to track replica state using EPHEMERAL nodes that disappear
automatically when a node goes down.*

Hi,

When running a cluster with many collections and replicas per node,
processing of DOWNNODE messages takes more time.
In a public cloud setup, the node that went down can come back quickly
before that processing is finished. When that happens, replicas are marked
DOWN by DOWNNODE while they are marked ACTIVE by the node starting, and
depending on how the two operations intermesh, some replicas then stay DOWN
forever (forever is until node is restarted).
We had to put in place K8s init containers to add a delay before nodes
restart. This delays rolling restarts, deployments and node crash recovery
so not a desirable long term solution.

What do you think of a change that avoids the need for a DOWNNODE message
altogether:
- Each replica state is captured as an *EPHEMERAL* node in Zookeeper
- No such node implicitly means the replica state is DOWN
- If the node is present, it contains an encoding of the actual state (DOWN,
ACTIVE, RECOVERING, RECOVERY_FAILED)
- When a node goes down (or when its ZK session expires) all its replica
state nodes automatically vanish.

This change is similar to the Per Replica State implementation (starting
point
<https://github.com/apache/solr/blob/main/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java#L99C17-L99C17>
in the code) but different:
- EPHEMERAL rather than PERSISTENT Zookeeper nodes
- No duplicate replica state nodes (and no node version to pick the right
one)
- DOWNNODE not needed (if all collections are tracked in that way).
- Need to republish all replica states after Zookeeper session expiration
since they will disappear

What do you think? esp. Noble and Ishan the authors of PRS.
I have no detailed design and no code, just sharing an idea to solve
a real issue we're facing.

Ilan

Re: Change suggestion: more efficient replica state tracking

[Ishan Chattopadhyaya <ic...@gmail.com>]

Thanks for bringing up this topic, Ilan.

> We are also running a fork of Solr and in our
> fork we have made some optimizations to avoid processing DOWNNODE
> messages for nodes that only host PRS collections. Those optimizations
> have not made it upstream at this point. I can take a look at
> upstreaming those changes or some variation of those.

Justin, my bad that I may have missed upstreaming some of the fixes. I can
take a look at consolidating all changes to upstream. I can see that
Fullstory's fork, upstream 8x and upstream 9x are all off by some fixes,
and I'm trying to consolidate them. I'm tracking the 8x branch mainly from
the 8.11 release standpoint, and shall attempt to bring all these branches
to sync such that whatever is working at scale inside Fullstory can be used
by everyone.

On Mon, 2 Oct 2023 at 16:52, Ilan Ginzburg <il...@gmail.com> wrote:

> Not sure I totally follow what you mean Mark.
> We thought making actual replica state = published replica state AND node
> state, which would set practical replica states to down when an ephemeral
> Zookeeper node for a SolrCloud node disappears.
> This works nicely for the going down part, but still requires work for
> coming back up, since at startup the SolrCloud node likely wants to
> advertise itself as up, but doesn't want all its replicas to appear as
> ACTIVE right away. So that would still require a loop updating all states
> for all replicas of the node to DOWN at startup, then updating them back to
> ACTIVE as they're becoming active.
>
> The main downside of EPHEMERAL replica state nodes in Zookeeper is the
> (added) bookkeeping required on ZK session expiration and reconnection to
> recreate all replica states.
> In our production clusters though, ZK session expiration is not correctly
> recovered from. The nodes with expired and reconnected sessions appear ok
> but their replicas are not really seen by the rest of the cluster (whatever
> that means, we didn't manage yet to understand the issue).
>
> I wonder if ZK session expiration and re establishment works nicely for
> others? The code handling this is in ZkController.onReconnect().
>
> On Mon, Oct 2, 2023 at 9:08 AM Mark Miller <ma...@gmail.com> wrote:
>
> > Actually, I think what I did was move the DOWN state to startup. Since
> you
> > can’t count on it on shutdown (crash, killed process, state doesn’t get
> > published for a variety of reasons), it doesn’t do anything solid for the
> > holes where you are indexing and a node cycles. So it can come up in any
> > state, like active, before it publishes recovery. So I kept it so a node
> > can set everything as down before publishing its live node.
> >
>

Re: Change suggestion: more efficient replica state tracking

[Ilan Ginzburg <il...@gmail.com>]

>
> I wonder if ZK session expiration and re establishment works nicely for
> others? The code handling this is in ZkController.onReconnect().


Answering my own question: the issue was specific to our fork so I assume
ZK session expiration and re establishment does work nicely in general.

On Mon, Oct 2, 2023 at 6:57 PM Mark Miller <ma...@gmail.com> wrote:

> Oh I’m not referring to your proposal, just what happens with the current
> system and what I had done around DOWN state to address it. The problem
> there is the missing replica state when you come up as the live node can’t
> cover for it. How you cover for that state could be done in a lot of ways,
> it’s just not covered currently.
>

Re: Change suggestion: more efficient replica state tracking

[Mark Miller <ma...@gmail.com>]

Oh I’m not referring to your proposal, just what happens with the current
system and what I had done around DOWN state to address it. The problem
there is the missing replica state when you come up as the live node can’t
cover for it. How you cover for that state could be done in a lot of ways,
it’s just not covered currently.

Re: Change suggestion: more efficient replica state tracking

[Ilan Ginzburg <il...@gmail.com>]

Not sure I totally follow what you mean Mark.
We thought making actual replica state = published replica state AND node
state, which would set practical replica states to down when an ephemeral
Zookeeper node for a SolrCloud node disappears.
This works nicely for the going down part, but still requires work for
coming back up, since at startup the SolrCloud node likely wants to
advertise itself as up, but doesn't want all its replicas to appear as
ACTIVE right away. So that would still require a loop updating all states
for all replicas of the node to DOWN at startup, then updating them back to
ACTIVE as they're becoming active.

The main downside of EPHEMERAL replica state nodes in Zookeeper is the
(added) bookkeeping required on ZK session expiration and reconnection to
recreate all replica states.
In our production clusters though, ZK session expiration is not correctly
recovered from. The nodes with expired and reconnected sessions appear ok
but their replicas are not really seen by the rest of the cluster (whatever
that means, we didn't manage yet to understand the issue).

I wonder if ZK session expiration and re establishment works nicely for
others? The code handling this is in ZkController.onReconnect().

On Mon, Oct 2, 2023 at 9:08 AM Mark Miller <ma...@gmail.com> wrote:

> Actually, I think what I did was move the DOWN state to startup. Since you
> can’t count on it on shutdown (crash, killed process, state doesn’t get
> published for a variety of reasons), it doesn’t do anything solid for the
> holes where you are indexing and a node cycles. So it can come up in any
> state, like active, before it publishes recovery. So I kept it so a node
> can set everything as down before publishing its live node.
>

Re: Change suggestion: more efficient replica state tracking

[Mark Miller <ma...@gmail.com>]

Actually, I think what I did was move the DOWN state to startup. Since you
can’t count on it on shutdown (crash, killed process, state doesn’t get
published for a variety of reasons), it doesn’t do anything solid for the
holes where you are indexing and a node cycles. So it can come up in any
state, like active, before it publishes recovery. So I kept it so a node
can set everything as down before publishing its live node.

Re: Change suggestion: more efficient replica state tracking

[Mark Miller <ma...@gmail.com>]

Yeah, I think a jira issue or two was filed for it, but I didn't see
anything user facing go in. You can do it for queries by asking the
overseer to publish a DOWN state though. It won't drop indexing leadership
until you close the core, but it will prevent the temporary slow/hotspot
you get if you shutdown a replica otherwise.

Re: Change suggestion: more efficient replica state tracking

[Houston Putman <ho...@apache.org>]

Yeah ive mentioned it a number of times, but it’s absolutely something we
should have. Give up leadership, dont accept new replicas, dont accept new
requests. Maybe remove live_node!

- Houstob

On Thu, Sep 28, 2023 at 6:45 PM David Smiley <da...@gmail.com>
wrote:

> Somewhat related to this, we don't yet have a way to signal to the cluster
> that a node will soon be shut down, but has not shut down yet.  With such
> information, we'd prefer to not send queries there, and perhaps could even
> begin shard/overseer leadership changes.  This was mentioned somewhere in
> connection with MIGRATE_REPLICAS but I don't see it.
>
> I wonder if manually removing the ephemeral ZK-node for the Solr host/node
> might accomplish this?
>
> ~ David
>
>
> On Thu, Sep 28, 2023 at 5:17 PM Mark Miller <ma...@gmail.com> wrote:
>
> > That did require some changes around live node handeling, which is why a
> > different approach as you suggest would also be reasonable. You still do
> > want to solve for the original motivation of DOWN - stopping search
> traffic
> > to the node before things start closing.
> >
>

Re: Change suggestion: more efficient replica state tracking

[David Smiley <da...@gmail.com>]

Somewhat related to this, we don't yet have a way to signal to the cluster
that a node will soon be shut down, but has not shut down yet.  With such
information, we'd prefer to not send queries there, and perhaps could even
begin shard/overseer leadership changes.  This was mentioned somewhere in
connection with MIGRATE_REPLICAS but I don't see it.

I wonder if manually removing the ephemeral ZK-node for the Solr host/node
might accomplish this?

~ David


On Thu, Sep 28, 2023 at 5:17 PM Mark Miller <ma...@gmail.com> wrote:

> That did require some changes around live node handeling, which is why a
> different approach as you suggest would also be reasonable. You still do
> want to solve for the original motivation of DOWN - stopping search traffic
> to the node before things start closing.
>

Re: Change suggestion: more efficient replica state tracking

[Mark Miller <ma...@gmail.com>]

That did require some changes around live node handeling, which is why a
different approach as you suggest would also be reasonable. You still do
want to solve for the original motivation of DOWN - stopping search traffic
to the node before things start closing.

Re: Change suggestion: more efficient replica state tracking

[Mark Miller <ma...@gmail.com>]

Yeah, I took the DOWN state out all together in shutdown as its problematic
and effectively sugar for the user view of the cluster state - as far as
the system goes, if the ephemeral live node is gone, that node is down,
regardless of the replica state. There is some value in being able to
remove an individual replica from search (rather than the full node), but
even that doesn't require the DOWN state publishing on shutdown.

Re: Change suggestion: more efficient replica state tracking

[Justin Sweeney <ju...@gmail.com>]

You are right, sorry. We are also running a fork of Solr and in our
fork we have made some optimizations to avoid processing DOWNNODE
messages for nodes that only host PRS collections. Those optimizations
have not made it upstream at this point. I can take a look at
upstreaming those changes or some variation of those.

I can definitely say that those of us doing Solr at FullStory would be
excited to continue to work together on improvements to PRS whether
that takes the form of ephemeral nodes or other improvements.

On Wed, Sep 27, 2023 at 9:29 AM Ilan Ginzburg <il...@gmail.com> wrote:
>
> Justin,
>
> Thanks for your reply. My understanding is that with PRS, DOWNNODE on
> Overseer still iterates over all collections and marks the relevant
> replicas down. It may be faster, but it's not a no-op. Did I miss something?
> We do not use PRS in our current setup.
>
> I agree with the risk related to ZK session expiration. TBH we already have
> quite a few issues around session expiration, we've observed that the
> replicas on that node are no longer correctly seen by the rest of the
> cluster and therefore no longer used. Didn't manage to root cause this yet
> so I don't know if it's related to local changes (we run a fork of upstream
> SolrCloud) or if it's a general problem.
> In any case, if PRS is modified to use EPHEMERAL ZK nodes, indeed ZK
> session expiration is something to be careful with.
>
> Ilan
>
> On Wed, Sep 27, 2023 at 1:24 AM Justin Sweeney <ju...@gmail.com>
> wrote:
>
> > Hey Ilan, curious if you have tried PRS in your implementation or not
> > at this point and what your experience has been if you have tried it?
> > I believe PRS currently publishes DOWNNODE messages to overseer, but
> > they are essentially a no-op by the overseer so they have very little
> > impact. We are running a cluster with many collections/shards and PRS
> > has been a huge improvement for us in processing nodes going down/up.
> >
> > The idea of ephemeral nodes seems interesting, but maybe some added
> > risk around Zookeeper session expiration and re-establishing replica
> > state.
> >
> > Justin
> >
> > On Tue, Sep 26, 2023 at 6:14 PM Ilan Ginzburg <il...@gmail.com> wrote:
> > >
> > > *TL;DR; a way to track replica state using EPHEMERAL nodes that disappear
> > > automatically when a node goes down.*
> > >
> > > Hi,
> > >
> > > When running a cluster with many collections and replicas per node,
> > > processing of DOWNNODE messages takes more time.
> > > In a public cloud setup, the node that went down can come back quickly
> > > before that processing is finished. When that happens, replicas are
> > marked
> > > DOWN by DOWNNODE while they are marked ACTIVE by the node starting, and
> > > depending on how the two operations intermesh, some replicas then stay
> > DOWN
> > > forever (forever is until node is restarted).
> > > We had to put in place K8s init containers to add a delay before nodes
> > > restart. This delays rolling restarts, deployments and node crash
> > recovery
> > > so not a desirable long term solution.
> > >
> > > What do you think of a change that avoids the need for a DOWNNODE message
> > > altogether:
> > > - Each replica state is captured as an *EPHEMERAL* node in Zookeeper
> > > - No such node implicitly means the replica state is DOWN
> > > - If the node is present, it contains an encoding of the actual state
> > (DOWN,
> > > ACTIVE, RECOVERING, RECOVERY_FAILED)
> > > - When a node goes down (or when its ZK session expires) all its replica
> > > state nodes automatically vanish.
> > >
> > > This change is similar to the Per Replica State implementation (starting
> > > point
> > > <
> > https://github.com/apache/solr/blob/main/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java#L99C17-L99C17
> > >
> > > in the code) but different:
> > > - EPHEMERAL rather than PERSISTENT Zookeeper nodes
> > > - No duplicate replica state nodes (and no node version to pick the right
> > > one)
> > > - DOWNNODE not needed (if all collections are tracked in that way).
> > > - Need to republish all replica states after Zookeeper session expiration
> > > since they will disappear
> > >
> > > What do you think? esp. Noble and Ishan the authors of PRS.
> > > I have no detailed design and no code, just sharing an idea to solve
> > > a real issue we're facing.
> > >
> > > Ilan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org

Re: Change suggestion: more efficient replica state tracking

[Ilan Ginzburg <il...@gmail.com>]

Justin,

Thanks for your reply. My understanding is that with PRS, DOWNNODE on
Overseer still iterates over all collections and marks the relevant
replicas down. It may be faster, but it's not a no-op. Did I miss something?
We do not use PRS in our current setup.

I agree with the risk related to ZK session expiration. TBH we already have
quite a few issues around session expiration, we've observed that the
replicas on that node are no longer correctly seen by the rest of the
cluster and therefore no longer used. Didn't manage to root cause this yet
so I don't know if it's related to local changes (we run a fork of upstream
SolrCloud) or if it's a general problem.
In any case, if PRS is modified to use EPHEMERAL ZK nodes, indeed ZK
session expiration is something to be careful with.

Ilan

On Wed, Sep 27, 2023 at 1:24 AM Justin Sweeney <ju...@gmail.com>
wrote:

> Hey Ilan, curious if you have tried PRS in your implementation or not
> at this point and what your experience has been if you have tried it?
> I believe PRS currently publishes DOWNNODE messages to overseer, but
> they are essentially a no-op by the overseer so they have very little
> impact. We are running a cluster with many collections/shards and PRS
> has been a huge improvement for us in processing nodes going down/up.
>
> The idea of ephemeral nodes seems interesting, but maybe some added
> risk around Zookeeper session expiration and re-establishing replica
> state.
>
> Justin
>
> On Tue, Sep 26, 2023 at 6:14 PM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > *TL;DR; a way to track replica state using EPHEMERAL nodes that disappear
> > automatically when a node goes down.*
> >
> > Hi,
> >
> > When running a cluster with many collections and replicas per node,
> > processing of DOWNNODE messages takes more time.
> > In a public cloud setup, the node that went down can come back quickly
> > before that processing is finished. When that happens, replicas are
> marked
> > DOWN by DOWNNODE while they are marked ACTIVE by the node starting, and
> > depending on how the two operations intermesh, some replicas then stay
> DOWN
> > forever (forever is until node is restarted).
> > We had to put in place K8s init containers to add a delay before nodes
> > restart. This delays rolling restarts, deployments and node crash
> recovery
> > so not a desirable long term solution.
> >
> > What do you think of a change that avoids the need for a DOWNNODE message
> > altogether:
> > - Each replica state is captured as an *EPHEMERAL* node in Zookeeper
> > - No such node implicitly means the replica state is DOWN
> > - If the node is present, it contains an encoding of the actual state
> (DOWN,
> > ACTIVE, RECOVERING, RECOVERY_FAILED)
> > - When a node goes down (or when its ZK session expires) all its replica
> > state nodes automatically vanish.
> >
> > This change is similar to the Per Replica State implementation (starting
> > point
> > <
> https://github.com/apache/solr/blob/main/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java#L99C17-L99C17
> >
> > in the code) but different:
> > - EPHEMERAL rather than PERSISTENT Zookeeper nodes
> > - No duplicate replica state nodes (and no node version to pick the right
> > one)
> > - DOWNNODE not needed (if all collections are tracked in that way).
> > - Need to republish all replica states after Zookeeper session expiration
> > since they will disappear
> >
> > What do you think? esp. Noble and Ishan the authors of PRS.
> > I have no detailed design and no code, just sharing an idea to solve
> > a real issue we're facing.
> >
> > Ilan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>

Re: Change suggestion: more efficient replica state tracking

[Justin Sweeney <ju...@gmail.com>]

Hey Ilan, curious if you have tried PRS in your implementation or not
at this point and what your experience has been if you have tried it?
I believe PRS currently publishes DOWNNODE messages to overseer, but
they are essentially a no-op by the overseer so they have very little
impact. We are running a cluster with many collections/shards and PRS
has been a huge improvement for us in processing nodes going down/up.

The idea of ephemeral nodes seems interesting, but maybe some added
risk around Zookeeper session expiration and re-establishing replica
state.

Justin

On Tue, Sep 26, 2023 at 6:14 PM Ilan Ginzburg <il...@gmail.com> wrote:
>
> *TL;DR; a way to track replica state using EPHEMERAL nodes that disappear
> automatically when a node goes down.*
>
> Hi,
>
> When running a cluster with many collections and replicas per node,
> processing of DOWNNODE messages takes more time.
> In a public cloud setup, the node that went down can come back quickly
> before that processing is finished. When that happens, replicas are marked
> DOWN by DOWNNODE while they are marked ACTIVE by the node starting, and
> depending on how the two operations intermesh, some replicas then stay DOWN
> forever (forever is until node is restarted).
> We had to put in place K8s init containers to add a delay before nodes
> restart. This delays rolling restarts, deployments and node crash recovery
> so not a desirable long term solution.
>
> What do you think of a change that avoids the need for a DOWNNODE message
> altogether:
> - Each replica state is captured as an *EPHEMERAL* node in Zookeeper
> - No such node implicitly means the replica state is DOWN
> - If the node is present, it contains an encoding of the actual state (DOWN,
> ACTIVE, RECOVERING, RECOVERY_FAILED)
> - When a node goes down (or when its ZK session expires) all its replica
> state nodes automatically vanish.
>
> This change is similar to the Per Replica State implementation (starting
> point
> <https://github.com/apache/solr/blob/main/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/PerReplicaStatesOps.java#L99C17-L99C17>
> in the code) but different:
> - EPHEMERAL rather than PERSISTENT Zookeeper nodes
> - No duplicate replica state nodes (and no node version to pick the right
> one)
> - DOWNNODE not needed (if all collections are tracked in that way).
> - Need to republish all replica states after Zookeeper session expiration
> since they will disappear
>
> What do you think? esp. Noble and Ishan the authors of PRS.
> I have no detailed design and no code, just sharing an idea to solve
> a real issue we're facing.
>
> Ilan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org